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Executive  Summary 


The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E)  was 
chartered  by  the  Deputy  Director,  Test,  Systems  Engineering  and  Evaluation  (Test  and 
Evaluation)*,  Office  of  the  Secretary  of  Defense  (OSD)  (Acquisition  and  Technology)  in  October 
1994  to  investigate  the  utility  of  advanced  distributed  simulation  (ADS)  technologies  for  support 
of  test  and  evaluation  (T&E).  The  JADS  program  was  Air  Force  led  with  Navy  and  Army 
participation.  This  report  addresses  the  overall  findings,  conclusions  and  recommendations  of 
JADS. 

JADS  directly  investigated  ADS  applications  in  three  slices  of  the  T&E  spectrum:  the  System 
Integration  Test  (SIT)  explored  ADS  support  of  precision  guided  munitions  (PGM)  testing;  the 
End-To-End  (ETE)  Test  investigated  ADS  support  for  command,  control,  communications, 
computers,  and  intelligence  (C4I)  testing;  and  the  Electronic  Warfare  (EW)  Test  evaluated  ADS 
support  for  EW  testing.  The  JADS  Joint  Test  Force  also  observed,  or  participated  at  a  modest 
level  in,  ADS  activities  sponsored  and  conducted  by  other  agencies.  The  purpose  of  this  effort 
was  twofold.  First  JADS  could  leverage  select  agencies  to  narrow  our  focus.  Second, 
involvement  with  other  agencies  using  this  technology  would  help  broaden  the  conclusions 
developed  in  our  three  dedicated  test  areas. 

Based  on  its  charter,  JADS  developed  questions  in  the  form  of  issues.  These  issues  were  used  to 

Table  ESI.  JADS  Issues  and  Objectives  Summary 


Issues 

Objectives 

Issue  1:  What  is  the  present 
utility  of  ADS,  including 
distributed  interactive  simulation 
(DIS),forT&E? 

Objective  1-1:  Assess  the  validity  of  data  from  tests  using  ADS, 
including  DIS,  during  test  execution. 

Objective  1-2:  Assess  the  benefits  of  using  ADS,  including  DIS,  in 
T&E. 

Issue  2:  What  are  the  critical 
constraints,  concerns,  and 

methodologies  when  using  ADS 
for  T&E? 

Objective  2-1:  Assess  the  critical  constraints  and  concerns  in  ADS 
performance  for  T&E. 

Objective  2-2:  Assess  the  critical  constraints  and  concerns  in  ADS 
support  systems  for  T&E. 

Objective  2-3:  Develop  and  assess  methodologies  associated  with 
ADS  for  T&E. 

Issue  3:  What  are  the 
requirements  that  must  be 
introduced  into  ADS  systems  if 
they  are  to  support  a  more 
complete  T&E  capability  in  the 
future? 

Objective  3-1:  Identify  requirements  for  ADS  systems  that  would 
provide  a  more  complete  T&E  capability  in  the  future. 

‘  This  office  is  now  the  Deputy  Director,  Developmental  Test  and  Evaluation. 
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structure  the  JADS  test  and  evaluation  approach.  JADS  then  developed  objectives  and  measures 
from  these  issues  to  guide  the  analysis.  The  first  issue  focused  on  assessing  the  utility  of  ADS 
for  T&E.  The  second  issue  focused  on  constraints,  concerns,  and  methodologies  when  using 
ADS  for  T&E.  And,  the  third  issue  focused  on  requirements  for  future  ADS  development  to 
improve  its  T&E  utility.  The  questions  raised  by  the  T&E  community  concerning  the  use  of 
ADS  technology  to  support  T&E  included:  “Will  it  work  with  the  rigor  necessary  to  support 
testing?”  “Are  the  data  gathered  using  ADS  valid  and/or  credible?”  “Is  the  technology  mature 
enough  for  a  T&E  tool?”  and  “How  affordable  is  using  this  technology  for  testing?”  JADS  was 
structured  to  answer  these  questions.  Table  ESI  summarizes  the  program-level  issues  and 
objectives  that  are  addressed  in  detail  in  Section  2  of  the  body  of  the  report. 

The  following  summarizes  the  answers  the  final  report  provides  to  the  three  issues. 

The  three  JADS  tests  each  demonstrated  the  utility  of  ADS  for  specific  classes  of  systems.  ADS 
architectures  can  provide  valid,  credible  data  and  can  therefore  be  used  in  support  of  rigorous 
T&E.  While  the  benefits  and  costs  of  ADS  are  very  program  specific,  JADS  data  strongly 
suggest  that  ADS  can  be  a  cost  effective  test  tool.  In  many  instances  the  technology  provides  for 
enhanced  testing  capability  at  reduced  costs. 

JADS  used  both  quantitative  and  qualitative  techniques  to  address  critical  concerns,  constraints 
and  methodologies.  JADS  experience  was  that  concerns  and  constraints  are  more  progranunatic 
than  technical.  JADS  assessed  network  and  communications  performance  in  three  key  areas: 
bandwidth,  latency  and  data  quality.  A  key  finding  of  JADS  was  that  in  most  cases  these  were 
indeed  only  concerns.  Network  and  communications  were  never  insurmountable  constraints  for 
the  JADS  tests  and  shouldn’t  be  for  future  tests  if  JADS  developed  test-planning  methodologies 
are  followed.  Programmatic  challenges  on  the  other  hand  are  many  and  diverse.  A  strong 
systems  engineering  approach  to  test  planning  and  design  is  critical  to  success. 

JADS  gathered  valuable  information  on  the  maturity  of  ADS  to  support  T&E  in  general.  Based 
on  JADS  experience  this  capability  is  mature  enough  to  support  most  applications  to  T&E.  In 
terms  of  the  requirements  to  provide  a  more  complete  T&E  capability  in  the  future,  ADS  is  still 
evolving  and  issues  remain  in  areas  such  as  fidelity,  data  structures,  security,  scalability, 
emission  representations  and  reactive  terrain. 


Lessons  Learned 

Near  the  end  of  the  JADS  program,  we  came  to  the  realization  that  the  term  “Advanced 
Distributed  Simulation  (ADS)”  was  potentially  misleading  when  used  in  the  context  of  T&E. 
The  word  “simulation”  is  something  of  a  turn  off  for  testers,  because  they  are  sharply  focused  on 
collecting  real  data  whenever  they  can.  In  fact,  distributed  architectures  can  be  used  effectively 
to  support  actual  testing.  A  system  under  test  (SUT)  can  be  incorporated  into  a  distributed 
architecture  and  subjected  to  valid  stimuli  that  can  originate  at  a  number  of  remote  sites.  The 
collection  of  stimuli  may  indeed  create  an  “artificial  environment”  which  is  perceived  by  the 
SUT,  but  if  the  stimuli  are  valid,  real  performance  data  can  be  collected  on  the  performance  of 
the  SUT.  That  is  testing.  (As  an  aside,  even  in  traditional  testing,  the  test  environment  is,  more 
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often  than  not,  artificial.)  The  fact  that  a  stimulus  originates  at  some  distance  from  the  SUT 
location  is  immaterial  so  long  as  it  is  a  valid  stimulus.  Distributed  architectures  can  provide 
robust  test  environments  and  offer  opportunities  to  conduct  concurrent,  rather  than  sequential  test 
events,  that  generate  actual  SUT  performance  data.  Distributed  testing,  as  we  see  it,  is  a  subset 
of  ADS.  There  will  almost  certainly  be  opportunities  to  use  ADS  as  a  simulation  tool  in  support 
of  acquisition  programs,  but  there  will  also  be  potential  benefits  to  using  distributed  architectures 
as  genuine  test  tools.  This  report  further  characterizes  lessons  learned  as  either  technical  or 
programmatic.  In  some  cases  the  lessons  learned  applied  to  more  than  one  category  and  thus 
appear  multiple  times.  For  each  category  an  attempt  was  make  to  distill  the  lessons  learned  in 
that  category  into  some  key  findings. 


Technical  Lessons  Learned 

-  Careful  design  and  thorough  integration  testing  are  required  regardless  of  whether  you’re 
developing  and  linking  a  new  simulation  or  linking  an  existing  stand-alone  simulation. 

-  For  the  foreseeable  future,  distributed  testing  must  rely  on  the  use  of  simulations  that  were 
not  designed  to  necessarily  work  together.  In  most  cases  it  is  more  cost  effective  to  develop 
interface  units  than  it  is  to  modify  existing  simulations.  Careful  design  and  testing  of  these 
interfaces  are  required  to  ensure  they  perform  desired  functions  without  adding  large 
processing  delays. 

-  Distributed  testing  applications  have  unique  and  sometimes  stringent  network  performance 
requirements  for  latency  and  latency  variation.  There  are  many  design  decisions  that  impact 
network  performance.  Special  instrumentation  and  thorough  testing  are  required  to  ensure 
the  network  is  meeting  performance  requirements. 

-  Special  network  instrumentation  is  required  to  monitor  network  performance  during  both 
development  and  test  execution.  Time  stamping  and  synchronization  are  unique  aspects  of 
distributed  testing  that  require  special  attention.  Live  players  also  have  unique 
instrumentation  requirements  for  distributed  test  applications. 

-  There  are  many  unique  aspects  to  distributed  test  control  that  require  a  carefully  selected  mix 
of  centralized  and  local  control.  The  central  control  facility  must  have  the  display  and 
communications  capabilities  to  know  total  system  health  in  real  time.  Total  system  health 
includes  not  only  the  status  of  the  real,  virtual,  and  constructive  players,  but  also  the  data 
processing  and  collection  systems  and  the  system  synchronization  mechanism.  Real-time 
processing  of  system  data  is  essential  to  efficient  test  conduct. 

-  Testing  in  software  quality  is  not  generally  possible.  Poorly  designed  software  rarely 
emerges  from  testing  in  any  better  condition.  Conversely  one  should  not  take  the  position 
that  well  planned  and  developed  software  does  not  require  testing.  The  development  and 
coordination  of  complex  models  to  support  distributed  testing  requires  extraordinary 
attention  to  configuration  management  issues.  The  added  complexity  of  distributed  testing 
over  stand-alone  applications  makes  following  proven  software  engineering  standards  and 
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procedures  critical.  Configuration  control  is  particularly  problematic  and  requires 
extraordinary  attention. 

—  Distributed  testing  requires  strong  systems  integration  and  systems  engineering  skills.  There 
are  important  considerations  in  the  planning  process  that  are  not  well  understood  and 
therefore  may  lead  to  unanticipated  costs.  This  report  sheds  light  on  many  of  these 
considerations.  There  are  two  unique  aspects  of  distributed  testing  that  affect  data  analysis 
and  must  be  planned  for  early.  The  first  is  the  requirement  to  do  real-time  data  analysis  to 
support  test  control  and  test  execution.  The  second  is  that  distributed  testing  generates  large 
amounts  of  data  in  a  short  time.  Without  careful  planning  and  testing  of  the  entire  data 
collection,  processing,  and  analysis  process,  test  analysts  will  be  either  hopelessly  lost  or 
hopelessly  buried  in  data. 

-  JADS  techniques  require  a  SUT  simulation  with  realistic  input-output  capabilities. 

-  Distributed-testing  architecture  can  provide  a  tool  for  operational  interoperability 
assessments  if  proper  fidelity  exists  for  each  simulation.  The  Navy’s  Distributed  Engineering 
Plant  (DEP)  concept  is  very  similar  to  JADS  in  placing  emphasis  on  interoperability  and 
realistic  stimuli.  DEP  appears  to  be  a  legacy  for  JADS. 

—  High  level  architecture  (HLA)  was  adequate  to  support  the  JADS  EW  Test,  but  it  required  a 
lot  of  tuning  and  trial  and  error  testing  on  the  part  of  JADS  and  Defense  Modeling  and 
Simulation  Organization  (DMSO).  Better  documentation  that  provides  insight  into  the  inner- 
workings  of  the  mntime  infrastructure  (RTI)  would  help.  Thorough  instrumentation  is 
required  to  track  and  isolate  problems  between  the  network  and  the  RTI.  The  T&E 
community  needs  to  actively  participate  in  the  Architecture  Management  Group  to  promote 
the  development  of  HLA  standards  and  products  that  meet  the  needs  of  the  T«&E  community. 


Programmatic  Lessons  Learned 

—  The  number  and  types  of  procedural  lessons  learned  JADS  identified  clearly  substantiate  the 
need  to  develop  standards  for  the  use  of  ADS. 

—  The  increased  complexity  of  a  distributed  test  can  result  in  a  small  to  medium  increase  in 
cost.  Actual  costs,  however,  are  application  specific.  Cost  drivers  include  synthetic 
environment  (SE)  complexity,  fidelity  requirements,  experienced  personnel  requirements, 
configuration  management,  and  network  interfaces.  If  a  new  or  modified  simulation  must  be 
developed  to  support  the  conduct  of  the  distributed  test,  costs  can  be  medium  to  high. 

—  Many  diverse  skills  are  required  to  support  distributed  testing.  A  system  integrator  with 
strong  systems  engineering  experience  and  adequate  empowerment  is  required  to  pull 
everything  together. 
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Recommendations 


JADS  has  identified  requirements  that  must  be  introduced  into  distributed  testing  systems  if  they 

are.  to  support  a  more  complete  T&E  capability.  The  following  recommendations  include 

requirements  for  distributed  testing  of  systems  that  would  help  improve  T&E  capability  in  the 

future. 

-  Address  distributed  testing  approaches  in  the  test,  and  evaluation  master  plan  (TEMP). 

-  Focus  on  the  ability  of  distributed  testing  to  overcome  any  identified  test  limitations. 

-  Program  managers  and  operational  test  agencies  (OTAs)  should  embrace  and  implement 
Simulation,  Test  and  Evaluation  Process  (STEP)  and  Simulation  Based  Acquisition  (SBA). 
Distributed  testing  is  an  enabling  technology  for  STEP  and  SBA. 

-  Use  the  integrated  product  team  (IPT)/integrated  product  and  process  development  (IPPD) 
process  to  facilitate  utilization  of  assets  across  phases  of  development  including  requirements 
definition,  engineering,  manufacture  and  development,  test  and  evaluation,  operations  and 
training. 

-  Mid-  and  upper-management  should  encourage/require  T&E  vision  beyond  the  scope  of 
individual  test  events  and  specific  systems. 

-  When  making  a  distributed  testing  go/no  go  decision,  compare  distributed  testing  costs  to  the 
costs  of  the  alternative  method(s). 

-  Use  JADS-developed  distributed  testing  cost  guidance  to  help  identify  the  optimal  mix  of 
distributed  testing  and  traditional  means  of  testing. 

-  Department  of  Defense  (DoD)  should  develop  infrastructure  to  reduce  the  costs  of  linking. 

-  Use  a  distributed  test  environment  over  the  life  of  a  program. 

-  Incorporate  distributed  testing  into  the  curricula  for  formal  T&E  and  acquisition  schools. 

-  DoD  should  nurture  groundbreaking  programs  such  as  Foundation  Initiative  2010  and  Joint 
Strike  Fighter. 

-  Use  JADS-developed  distributed  testing  methodologies  such  as  test  planning,  verification 
and  validation. 

-  Each  organization  using  distributed  testing  should  plan  for  a  centralized  test  control  and 
analysis  capability.  Such  a  facility  can  be  low  cost  and  located  anywhere.  In  addition  to  test 
control,  it  can  be  used  to  enhance  real-time  data  analysis  and  test  efficiency. 
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Require  the  PMs  provide  a  realistic  HWIL  simulation  of  the  SUT  with  realistic 
representation  of  voice,  data,  and  data-links. 
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1.0  JADS  Overview 


1.1  Background 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E)  was 
chartered  by  the  Deputy  Director,  Test,  Systems  Engineering  and  Evaluation  (Test  and 
Evaluation)^,  Office  of  the  Secretary  of  Defense  (OSD)  (Acquisition  and  Technology)  in  October 
1994  to  investigate  the  utility  of  ADS  technologies  for  support  of  developmental  test  and 
evaluation  (DT&E)  and  operational  test  and  evaluation  (OT&E).  The  program  was  Air  Force  led 
with  Army  and  Navy  participation.  Science  Applications  International  Corporation  (SAIC)  and 
the  Georgia  Tech  Research  Institute  provided  contracted  technical  support. 

1.2  Purpose 

The  JADS  JT&E  charter  focused  on  three  issues:  What  is  the  present  utility  of  ADS,  including 
distributed  interactive  simulation  (DIS),  for  test  and  evaluation  (T&E);  What  are  the  critical 
constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E;  and  what  are  the 
requirements  that  must  be  introduced  into  ADS  systems  if  they  are  to  support  a  more  complete 
T&E  capability  in  the  future.  The  JADS  Joint  Test  Force  (JTF)  directly  investigated  ADS 
applications  in  three  slices  of  the  T&E  spectrum:  the  System  Integration  Test  (SIT)  explored 
ADS  support  of  air-to-air  missile  testing;  the  End-to-End  (ETE)  Test  investigated  ADS  support 
for  command,  control,  communications,  computers,  intelligence,  surveillance  and  reconnaissance 
(C4ISR)  testing;  and  the  Electronic  Warfare  (EW)  Test  evaluated  ADS  support  for  EW  testing. 
Each  test  applied  the  JADS  objectives  and  measures  as  appropriate  to  conduct  its  evaluation. 
The  JTF  was  also  chartered  to  observe  or  participate  at  a  modest  level  in  ADS  activities 
sponsored  and  conducted  by  other  agencies  in  an  effort  to  help  narrow  the  focus  and  broaden 
conclusions  developed  in  our  three  direct  test  areas. 

The  following  is  a  synopsis  of  each  of  the  JADS  distributed  tests. 

The  Sir  explored  the  utility  of  using  ADS  to  support  cost-effective  testing  of  an  integrated 
missile  weapon/launch  aircraft  system  in  an  operationally  realistic  scenario.  The  SIT  was  a 
Distributed  Interactive  Simulation  (DIS)  -based  test  and  consisted  of  two  phases,  each  of  which 
culminated  in  three  flight  missions.  The  missions  simulated  a  single  shooter  aircraft  launching 
an  air-to-air  missile  against  a  single  target  aircraft.  In  the  Linked  Simulators  Phase  (LSP),  the 
shooter,  target,  and  missile  were  all  represented  by  simulators.  In  the  Live  Fly  Phase  (LFP),  the 
shooter  and  target  were  represented  by  live  aircraft  and  the  missile  by  a  simulator. 

The  EW  Test  evaluated  the  utility  of  ADS  in  an  EW  environment.  The  first  distributed  test 
phase  employed  a  linked  architecture  using  Department  of  Defense’s  (DoD)  high  level 
architecture  (HLA)  which  included  a  digital  simulation  model  of  the  ALQ-131  self-protection 
jammer,  threat  simulation  facilities,  and  constructive  models  that  supported  replication  of  the 
open  air  environment.  In  the  second  phase,  an  installed  systems  test  facility  (ISTF)  was 


^  This  office  is  now  the  Deputy  Director,  Developmental  Test  and  Evaluation. 
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substituted  for  the  digital  model.  In  both  distributed  test  architectures,  system  performance  data 
were  compared  with  live  fly  data  for  verification  and  validation  (V&V). 

The  ETE  Test  investigated  the  utility  of  ADS  to  support  testing  of  C4ISR  systems.  It  conducted 
its  T&E  utility  evaluation  in  a  DIS-based,  ADS-enhanced  environment  using  the  Joint 
Surveillance  Target  Attack  Radar  System  (Joint  STARS)  as  one  component  of  a  representative 
C4ISR  environment. 

The  ETE  Test  used  ADS  to  assemble  an  enhanced  environment  for  testing  C4ISR  systems.  The 
intent  was  to  provide  a  complete,  robust  set  of  interfaces  from  sensor  to  weapon  system, 
including  the  additional  intermediate  nodes  that  would  be  found  in  a  tactical  engagement.  The 
test  traced  a  thread  of  the  complete  battlefield  process  from  target  detection  to  target  assignment 
and  engagement  at  corps  level  using  distributed  testing.  It  also  allowed  the  tester  to  evaluate  the 
thread  as  a  whole  or  the  contribution  of  any  of  the  parts  individually  and  to  evaluate  what  effects 
an  operationally  realistic  environment  had  on  the  system  under  test. 

The  ETE  Test  was  designed  to  add  additional  entities  in  a  seamless  manner  to  the  battlefield  seen 
by  Joint  STARS.  In  addition,  adding  some  of  the  complementary  suite  of  other  command, 
control,  communications,  computers  and  intelligence  (C4I)  and  weapon  systems  with  which 
Joint  STARS  would  interact  enabled  the  test  team  to  evaluate  the  utility  of  a  ADS-enhanced 
environment. 

All  three  tests  evaluated  the  capability  of  the  JADS  Test  Control  and  Analysis  Center  (TCAC)  to 
control  a  distributed  test  of  this  type  and  to  remotely  monitor  and  analyze  test  results. 

1.3  Test  and  Evaluation  Approach 

1.3.1  Test  Issues  and  Objectives 

Based  on  its  charter,  JADS  developed  questions  in  the  form  of  issues.  These  issues  were  then 
used  to  structure  the  JADS  test  and  evaluation  approach.  JADS  then  developed  objectives  and 
measures  from  these  issues  to  guide  the  analysis.  The  first  issue  focused  on  assessing  the  utility 
of  ADS  for  T«fcE.  The  second  issue  focused  on  constraints,  concerns,  and  methodologies  when 
using  ADS  for  T&E.  And,  the  third  issue  focused  on  requirements  for  future  ADS  development 
to  improve  its  T&E  utility.  The  questions  raised  by  the  T&E  community  concerning  the  use  of 
ADS  teehnology  to  support  T&E  included:  “Will  it  work  with  the  rigor  necessary  to  support 
testing?”  “Are  the  data  gathered  using  ADS  valid  and/or  credible?”  “Is  the  technology  mature 
enough  for  a  T&E  tool?”  and  “How  affordable  is  using  this  technology  for  testing?”  JADS  was 
structured  to  answer  these  questions.  Table  1  summarizes  the  program-level  issues  and 
objectives  described  in  detail  below.  The  subobjectives  and  measures  used  to  evaluate  these 
issues  and  objectives  are  addressed  in  Section  2. 
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Table  1.  JADS  Issues  and  Objectives  Summary 


Issues 

Objectives 

Issue  1 :  What  is  the  present  utility  of  ADS, 
including  DIS,  for  T&E? 

Objective  1-1:  Assess  the  validity  of  data  from 
tests  using  ADS,  including  DIS,  during  test 
execution. 

Objective  1-2:  Assess  the  benefits  of  using  ADS, 
including  DIS,  in  T&E. 

Issue  2:  What  are  the  critical  constraints, 
concerns,  and  methodologies  when  using 
ADS  for  T&E? 

Objective  2-1:  Assess  the  critical  constraints  and 
concerns  in  ADS  performance  for  T&E. 

Objective  2-2:  Assess  the  critical  constraints  and 
concerns  in  ADS  support  systems  for  T&E. 

Objective  2-3:  Develop  and  assess 

methodologies  associated  with  ADS  for  T&E. 

Issue  3:  What  are  the  requirements  that 
must  be  introduced  into  ADS  systems  if 
they  are  to  support  a  more  complete  T&E 
capability  in  the  future? 

Objective  3-1:  Identify  requirements  for  ADS 
systems  that  would  provide  a  more  complete 
T&E  capability  in  the  future. 

ISSUE  1:  What  is  the  present  utility  of  ADS,  including  DIS,  for  T&E? 

Some  questions  from  the  testing  community  concerning  the  use  of  ADS  to  support  T&E  were: 
“What  does  it  cost?”  “Does  it  work?”  “Will  it  support  T&E  earlier  in  the  acquisition  process?” 
To  be  useful  for  T&E,  ADS  must  either  provide  operational  realism  equivalent  to  live  testing  at 
reduced  cost,  or  it  must  provide  increased  operational  realism  at  an  affordable  cost.  Both  the 
costs  and  benefits  of  using  ADS  are  important  measures  to  determine  the  utility  of  distributed 
testing  to  support  T&E.  Two  objectives  were  used  to  address  this  issue. 

Objective  1-1:  Assess  the  validity  of  data  from  tests  using  ADS,  including  DIS,  during  test 
execution. 

The  key  to  the  utility  of  distributed  testing  for  T&E  lies  in  its  ability  to  provide  valid  data  when 
used  during  test  execution.  If  ADS  does  not  provide  valid  data  during  test  execution,  then  it  has 
no  utility.  If  it  does  provide  valid  data,  then  it  may  have  a  great  deal  of  utility. 
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Objective  1-2:  Assess  the  benefits  of  using  ADS,  including  DIS,  in  T&E. 

Once  the  validity  of  ADS  data  in  test  execution  was  established,  the  benefits  of  using  ADS  in 
T&E  were  addressed.  The  benefits  of  ADS  for  all  phases  of  T&E  as  well  as  for  the  early  phases 
of  the  acquisition  process  were  addressed  in  the  subobjectives  and  measures  for  this  objective. 

ISSUE  2:  What  are  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS  for 
T&E? 

This  issue  looked  at  characteristics  such  as  fidelity  and  maturity  of  the  technology  required  for 
T&E.  The  execution  of  a  test  will  determine  if  the  necessary  maturity  does  exist  as  well  as 
identify  the  strengths  and  weaknesses  in  the  maturity  of  the  technologies  for  other  T&E  use. 
JADS  used  three  objectives  to  address  this  issue. 

Objective  2-1:  Assess  the  critical  constraints  and  concerns  in  ADS  performance  for  T&E. 

In  the  design  of  a  specific  ADS  T&E  methodology  to  support  a  test,  assembling  a  network  of 
constructive,  virtual,  and  live  simulations  presents  a  host  of  new  technical  and  management 
issues  for  testers.  These  issues  may  include  such  problems  as  simulation  identification  and 
capability  evaluation,  integration  of  models,  interface  development  and  testing,  network 
development,  network  operations  and  scheduling  management,  and  verification  and  validation  of 
entire  networks.  In  addition,  factors  such  as  seamlessness,  fidelity,  and  latency  need  to  be 
considered  based  on  the  desired  output  or  function  of  the  ADS  T&E  methodology.  These  design 
issues  as  well  as  the  performance  shortfalls  of  the  resulting  implementation  were  addressed  in 
this  objective.  This  objective  also  looked  at  the  reliability  of  the  ADS  network  assembled  for 
each  test  program.  While  each  individual  component  in  a  ADS  network  has  its  own  reliability, 
the  linked  distributed  testing  network  of  simulations  and  live  systems  used  in  a  test  may  have  a 
totally  different  reliability.  This  was  an  important  maturity  issue  JADS  addressed  during  the 
JT&E.  Can  the  ADS  network  infrastructure  be  reliably  scheduled?  Will  the  network  run  when 
initiated?  Will  it  operate  continuously  for  an  adequate  amount  of  time  to  complete  the  test 
event?  The  ADS  T&E  methodology  infrastructure  must  possess  some  adequate  level  of 
operational  reliability  to  be  useful  to  support  the  T&E. 

Objective  2-2:  Assess  the  critical  constraints  and  concerns  in  ADS  support  systems  for  T&E. 

This  objective  addresses  the  maturity  of  the  overall  ADS  support  infrastructure.  The  impact,  as 
well  as  the  increased  use  of  simulation,  that  the  distributed  nature  of  ADS  testing  has  upon 
existing  configuration  management  systems  and  data  management  and  analysis  support  systems 
was  addressed. 


10 


Objective  2-3:  Develop  and  assess  methodologies  associated  with  ADS  for  T&E. 

The  JTF  modified  existing  procedures  as  necessary  or  developed  additional  procedures  for 
planning,  designing,  testing,  and  operating  ADS  methodologies  used  in  the  JT&E.  JADS  also 
identified  the  strengths  and  weaknesses  of  the  various  management,  network,  and  simulation 
issues  related  to  assembling  an  ADS  infrastructure.  To  help  determine  the  feasibility  and  utility 
of  ADS  test  methods,  JADS  published  A  Test .  Planning  Methodology  -  From  Concept 
Development  Through  Test  Execution  which  can  be  found  at  www.jads.abq.com.^ 

ISSUE  3:  What  are  the  requirements  that  must  be  introduced  into  ADS  systems  if  they  are  to 
support  a  more  complete  T&E  capability  in  the  future? 

The  test  community  has  not  yet  developed  its  requirements  for  specialized  support  in  the  form  of 
network  and  simulation  standards  to  support  T&E.  ADS  is  still  evolving  and  issues  remain  in 
areas  such  as  fidelity,  data  structures,  security,  scalability,  emission  representations,  and  reactive 
terrain.  Objective  3-1  was  used  to  address  this  issue. 

Objective  3-1:  Identify  requirements  for  ADS  systems  that  would  provide  a  more  complete  T&E 
capability  in  the  future. 

For  any  given  application  of  ADS  in  a  specific  test  program,  there  are  technological  alternatives 
for  implementation.  For  example,  only  one  of  several  networks  (e.g..  Defense  Simulation 
Internet  [DSI]  or  leased  commercial)  may  be  employed  during  a  given  test,  but  networks  may  not 
be  equally  capable  of  supporting  that  test.  In  the  development  of  the  test  concepts  for  this  JT&E, 
alternatives  were  considered  and  some  were  rejected  as  not  appearing  mature  enough  to  support 
the  test  during  its  life  span.  Also,  to  scope  this  JT&E  at  a  reasonable  level  of  cost,  choices  were 
made  concerning  which  systems  and  which  test  issues  would  be  selected  for  evaluation.  For 
these  reasons,  a  complete  assessment  of  the  maturity  of  this  technology  as  a  whole  cannot  be 
made  based  solely  on  JADS  results.  However,  JADS  gathered  valuable  information  on  the 
maturity  of  ADS  to  support  T&E  in  general.  Where  appropriate  for  this  objective,  this  report 
addresses  ADS  maturity  shortfalls  that  can  be  used  as  requirements  to  influence  future 
development  of  the  technology  to  support  T&E. 

1.3.2  Schedule 

JADS  basic  charter  was  granted  in  October  1994;  the  EW  Test  was  chartered  in  August  1996. 
The  JT&E  program  deactivates  in  March  2000.  The  program  schedule  for  the  major  activities 
within  each  test  program  and  the  significant  program  milestones  by  fiscal  year  (FY)  are  shown  in 
Figure  1.  As  illustrated,  multiple  tasks  for  each  of  the  test  programs  were  accomplished 
simultaneously. 


^  After  1  March  2001,  refer  requests  to  the  Joint  Program  Office  Technical  Library,  2001  North 
Beauregard  St.,  Suite  800,  Alexandria,  Virginia  22311. 
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JADS 

r&E  Summary  Schedule 

Task  Name 

FY95 

FY96 

FY97 

FY98 

FY99 

FYOO 

JT&E  Charter 

♦ 

♦ 

- 

Program  Test  Plan 

Legacy  Products 

< - 

IHHi 

Systems  Integration  Test 

Linked  Simulator  Phase 

◄ - 

Live  Fly  Phase 

End-to-End  Test 

◄ - 

- ► 

Phase  1 

◄ - 

-► 

.  ^ 

Phase  2 

w 

Phase  3 

4 - 

Phase  4 

Electronic  Warfare  Test 

< - 

♦ 

Phase  1 

Open  Air  Range 

♦ 

♦ 

♦ 

♦ 

Hardware-in-the-Loop 

System  Integration  Laboratory 

Phase  2  Digital  System  Model 
Test 

Phase  3  Installed  Systems  Test 
Facility  Test 

Final  Report 

Figure  1.  JADS  JT&E  Summary  Schedule 


1.4  Execution  Results 

This  section  provides  an  overview  of  each  test.  It  includes  a  summary  of  each  test,  test  results 
and  conclusions,  and  observations  for  general  classes  of  weapon  systems  that  are  based  upon  an 
extrapolation  of  JADS  results.  A  detailed  discussion  of  the  results  from  each  test  can  be  found  in 
the  specific  test’s  phase  and  final  reports. 

1.4.1  SIT  Executive  Summary 

The  SIT  explored  the  ability  of  ADS  to  support  air-to-air  missile  distributed  testing.  Two 
sequential  phases,  a  Linked  Simulators  Phase  (LSP)  and  a  Live  Fly  Phase  (LFP),  incorporated 
one-versus-one  scenarios  based  upon  profiles  flown  during  live  test  activities  and  limited  target 
countermeasure  capability. 
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SHOOTER 


MISSILE 


A/C  =  aircraft 

BMIC  =  Battle  management  Interoperability  Center 
SIM  =  simulation 
TGT  =  target 

WSSF  =  Weapon  System  Support  Facility 


AFB  =  Air  Force  Base 
IR  =  infrared 

SIMLAB  =  Simulation  Laboratory 
WSIC  =  Weapons  System  Integration  Center 


AIM  =  air  intercept  missile 
MSL  =  missile 

SMS  =  stores  management  system 


Figure  2.  SIT  LSP  Architecture 

The  LSP  distributed  architecture  incorporated  four  nodes:  the  shooter,  an  F/A-18  manned 
avionics  laboratory  at  China  Lake,  California;  the  target,  an  F-14  manned  avionics  laboratory  at 
Point  Mugu,  California;  a  hardware-in-the-loop  (HWIL)  missile  laboratory  at  China  Lake  that 
hosted  an  air  intercept  missile  (AIM)-9M  missile;  and  a  test  control  center  initially  located  at 
Point  Mugu  and  later  relocated  to  the  JADS  facility  in  Albuquerque,  New  Mexico. 

The  LFP  distributed  architecture  linked  two  live  F-16  aircraft  (a  shooter  and  target)  on  the  Eglin 
Air  Force  Base,  Florida,  Gulf  Test  Range;  the  Eglin  Central  Control  Facility;  an  HWIL  missile 
laboratory  at  Eglin  that  hosted  an  AIM- 120  missile;  and  a  test  monitoring  center  at  the  JADS 
facility  in  New  Mexico. 
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LIVE  AIRCRAFT 


MISSILE 


A/C  =  aircraft  A  AS  I  =  Advanced  Aircraft  Simulation  Interface 

AFB  =  Air  Force  Base  AMRAAM  =  advanced  medium  range  air-to-air  missile 

ECM  =  electronic  countermeasure  GPS  =  global  positioning  system  INS  =inertial  navigation  system 

MISILAB  =  Missile  Simulation  Laboratory  MSL  =  missile  RDL  =  rear  data  link 

RF  =  radio  frequency 

T-1  =  digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1.544  megabits  per  second 

T-3  =  28  T-1  lines  in  one;  the  aggregate  data  rate  is  44.746  megabits  per  second  TM  =  telemetry 

TSPl  =  time-space-position  information  UMB  =  umbilical 

Figure  3.  SIT  LFP  Architecture 


The  following  section  describes  the  outcome  of  the  SIT,  the  conclusions  and  lessons  learned,  and 
offers  observations  on  the  implications  of  SIT  for  the  general  class  of  precision  guided 
munitions. 


1.4.1. 1  System  Integration  Test  Results  and  Conclusions 

Within  the  narrow  confines  of  the  SIT  data,  our  assessment  is  that  the  two  architectures  we 
employed  have  utility  for  support  of  T&E.  The  JADS  data  indicate  that  activities  ranging  from 
parametric  analyses  to  integrated  weapons  system  testing  are  both  practical  and  cost  effective. 
Our  broad  conclusions  and  lessons  learned  can  be  summarized  as  follows. 

-  For  T&E  applications,  the  technology  is  not  at  the  “plug-and-play”  stage.  While  practical 
and  cost  effective  in  many  cases,  implementation  is  more  challenging  than  many  people 
think.  Plan  for  a  lot  of  rehearsals  and  “fix”  time. 


-  The  effects  of  latency  and  other  ADS-induced  errors  can  often  (not  always)  be  mitigated. 
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-  Synchronization  is  as  much  a  challenge  as  latency. 

-  Instrumentation  and  data  management  are  challenges. 

-  ADS  has  great  potential  as  a  T&E  support  tool:  It  is  a  valuable  addition  to  the  tester’s  tool 
kit.  ADS  will  not  obviate,  but  in  some  cases  it  may  reduce,  the  need  for  live  testing. 

-  Our  data  suggest  test  savings  are  possible. 

1.4.1.2  Observations  for  Precision  Guided  Munitions  T&E 

Through  a  process  of  inductive  reasoning  we  can  transfer  some  of  the  SIT-based  specifics  to  the 
general  class  of  precision  guided  munitions  (PGM).  In  the  general  case,  the  elements  of  the  SIT 
architectures  are  basic  to  all  PGM  cases.  There  are  (1)  a  launch  platform  or  shooter,  (2)  a  PGM, 
(3)  an  intended  target,  (4)  an  operating  environment  (to  include  countermeasures),  and  (5)  a  test 
control  center. 

The  shooter,  PGM,  and  target  can  be  represented  in  any  of  the  three  forms  associated  with 
distributed  simulation:  live,  virtual,  or  constructive.  SIT  looked  at  an  AIM-9  and  an  AIM-120. 
The  physical  dynamics  of  the  problem  are  comparable  with  any  class  of  PGM.  The  physics 
associated  with  detection,  tracking,  and  guidance  may  differ  significantly  depending  upon  bands, 
techniques,  and  the  operational  medium  a  missile  operates  in.  We  do  not  see  a  one-for-one 
transfer  of  SIT  techniques  to  other  tests.  Each  test  has  specific  requirements,  often  specific  to  the 
particular  system  under  test  (SUT).  We  do  see  a  transfer  of  the  principles,  design  processes,  and 
methodologies  used  in  SIT. 

Countermeasures  were  only  represented  in  rudimentary  form  in  the  SIT,  but  we  see  no  technical 
impediments,  at  the  conceptual  level,  to  implementing  high-fidelity  countermeasures  in 
distributed  testing.  The  difficulty  will  be  in  the  details,  and  costs  and  technical  challenges  will 
be  very  case  specific.  Complex  environmental  details  associated  with  atmospherics,  space, 
oceanography,  etc.,  are  more  challenging.  The  LFP,  since  it  involved  flying  open  air, 
incorporated  real  atmospheric  effects. 

A  test  control  center  is  a  requirement  for  all  testing,  distributed  or  not.  Fortunately,  the  SIT 
experience  suggests  that  the  control  center  can  function  from  almost  anywhere.  The  inference  is 
that  an  existing  control  center  somewhere  may  well  meet  a  specific  tester’s  needs. 

The  Srr  program  was  budget  and  schedule  constrained.  Consequently,  there  were  important 
aspects  of  PGM  testing  that  SIT  did  not  explore.  From  a  single  shooter  perspective,  some  of 
these  included  multiple  launches  against  a  single  target,  single  launches  against  clustered  targets, 
and  multiple  launches  against  multiple  targets.  SIT  did  not  examine  few-on-few  or  many-on- 
many  scenarios.  Our  expectation,  unsupported  by  hard  data,  is  that  few-on-few  implementations 
are  possible.  The  difficulties  and  costs  would  be  extremely  sensitive  to  the  fidelity  requirements 
and  the  availability  of  existing  facilities,  e.g.,  HWIL  facilities  or  installed  systems  test  facilities. 
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The  SIT  results  strongly  suggest  that  ADS  has  good  potential  for  improving  PGM  testing.  The 
implication  is  that  test  planners  should  consider  the  technology  as  a  relevant  tool  for  their 
program  until  an  objective  assessment  suggests  otherwise.  Bottom  line:  Know  distributed  testing 
is  there,  and  assess  how,  or  if,  it  should  be  used  in  a  specific  program. 

1.4.2  ETE  Test  Executive  Summary 

The  ETE  Test  evaluated  the  utility  of  ADS  to  support  mission-level  testing  of  C4ISR  systems. 
The  test  used  the  Joint  STARS  as  one  component  of  a  representative  C4ISR  system.  Other 
C4ISR  elements  represented  in  the  ETE  Test  included  virtual  battlefield  entities  (including  about 
10,000  threat  entities),  manned  air  and  ground  operator  workstations,  an  actual  Army  target 
analysis  cell,  a  fire  direction  center,  and  simulated  missiles  for  attacking  selected  targets. 
Tactical  communications  systems  were  used  between  most  of  these  elements.  Figure  4  shows 
what  a  typical  C4ISR  ADS  architecture  might  look  like  using  a  representation  of  the  JADS  ETE 
Test  configurations. 


Figure  4.  C4ISR  ADS  Architecture 


There  were  two  separate  representations  for  the  Joint  STARS  E-8C  aircraft.  In  the  laboratory 
configuration,  the  Virtual  Surveillance  Target  Attack  Radar  System  (VSTARS)  radar  emulation 
represented  the  E-8C  radar  subsystem  and  provided  the  inputs  needed  to  drive  target  displays  on 
operator  workstations.  In  the  live  configuration,  an  actual  E-8C  aircraft  was  flown  and  the 
workstation  operators  observed  actual  live  ground  targets  participating  in  an  exercise  along  with 
the  virtual  battlefield  entities. 

1. 4.2.1  End-to-End  Test  Results  and  Conclusions 

Within  the  confines  of  the  ETE  Test  data,  our  assessment  is  that  the  architecture  we  employed 
has  utility  for  support  of  C4ISR  T&E,  especially  realistic  mission-level  testing.  The  JADS  data 
indicate  that  DT&E  and  OT&E  activities  incorporating  ADS  technology  are  both  practical  and 
cost  effective. 
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-  ADS  often  requires  linkage  among  dissimilar  facilities,  network  equipment,  and  simulations. 
While  practical  and  cost  effective  in  many  cases,  implementation  is  more  challenging  than 
many  people  think.  Plan  for  a  number  of  rehearsals  and  periods  of  “fix”  time. 

-  ADS  requirements  must  be  clearly  defined  early  in  the  test  planning  phase,  since  individual 
facilities  are  generally  unfamiliar  with  conducting  coordinated,  distributed  T&E  tests. 

-  Instrumentation  and  data  management  are  challenges. 

-  Have  a  centralized  test  control  center  with  test  controllers  who  are  extremely  familiar  with 
the  test  and  network  configuration. 

-  ADS  testing  of  C4ISR  systems  is  technically  feasible,  provides  valid  data,  and  is  cost 
effective  in  many  cases. 

-  ADS  has  great  potential  as  a  C4ISR  testing  tool  and  provides  a  useable  means  of  conducting 
realistic  mission-level  evaluations. 

1.4.2.2  Observations  for  C4ISR  T&E 

Since  all  C4ISR  systems  contain  the  same  basic  elements  (e.g.,  sensor(s),  sensor  platform(s), 
command  and  control  elements,  communication  lines,  and  computer  hardware  and  software),  the 
extension  of  the  ETE  Test  results  to  other  possible  C4ISR  applications  is  relatively 
straightforward.  ADS  technology  allows  the  evaluation  of  human-in-the-loop  actions,  decision 
processes,  timelines,  and  interoperability  that  digital  simulations  do  not  model  well.  Using  ADS, 
a  mission-level  scenario  model  can  be  linked  to  actual  C4ISR  hardware  and  software  with 
tactical  operators-in-the-loop  and  tactical  communications  links  for  realistic  testing  in  force-on- 
force  scenarios  that  cannot  be  accomplished  in  live  testing. 

A  test  control  center  is  a  requirement  for  all  testing,  distributed  or  not.  Fortunately,  the  ETE  Test 
experience  suggests  that  the  control  center  can  function  from  almost  anywhere  with  costs 
tailored  to  the  test  requirements.  Thus,  a  specific  test  may  save  resources  by  using  an  existing 
control  center. 

ADS  is  not  just  of  value  to  C4ISR  T&E  but  can  be  applied  throughout  the  system  acquisition  life 
cycle.  In  fact,  the  benefits  of  using  ADS  are  best  realized  over  the  life  of  a  program.  ADS  is  an 
enabling  technology  for  Simulation  Based  Acquisition  (SBA)  and  the  Simulation,  Test  and 
Evaluation  Process  (STEP)  as  applied  to  C4ISR  systems,  since  these  systems  are  naturally 
distributed  by  nature. 

-  ADS  can  create  a  cost-effective  environment  for  test  preparation  to  include  the  development 
of  concepts  of  operations,  refinement  of  test  scenarios,  rehearsal  of  test  execution,  and  data 
collection  and  analysis. 

-  ADS  allows  the  integration  of  models  developed  by  different  acquisition  programs. 
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—  ADS  can  expedite  the  association  of  results  from  live  tests  with  output  of  simulations. 

-  supports  the  execution  of  the  Joint  Vision  2010  paradigm,  which  requires  realistic 
battle-space  environments  populated  with  many  weapon  systems  and  threats.  In  particular, 
ADS  allows  the  large-scale,  complex  environment  evaluations  needed  for  C4ISR  systems. 

-  ADS  can  support  the  model-test-model  process  by  providing  more  realistic  test  results  that 
can  be  used  to  refine  digital  system  models. 

-  ADS  enables  the  linking  and  integration  of  geographically  distributed  resources  from 
different  system  representation  domains  (e.g.,  digital  system  model,  hardware-in-the-loop 
laboratory,  integrated  systems  test  facility,  open  air  range)  that  can  lower  testing  costs. 

—  ADS  supports  experimentation  of  emerging  war  fighting  concepts  and  testing  new  weapon 
systems. 

—  ADS  can  reduce  the  operations  tempo  (OPTEMPO)  of  test  participants  and  evaluators  in 
pretest  training  and  integration  periods  as  well  as  in  the  test  period  itself. 

The  ETE  Test  results  strongly  suggest  that  ADS  has  excellent  potential  for  improving  C4ISR 
testing  and  system  acquisition.  Test  planners  and  program  managers  should  consider  the 
technology  as  a  relevant  tool  for  their  program  unless  an  objective  assessment  suggests 
otherwise. 

1.4.3  EW  Test  Executive  Summary 

The  EW  Test  evaluated  the  utility  of  ADS  to  support  EW  T&E.  While  the  test  used  several 
efforts  to  examine  distributed  testing-based  T&E,  the  cornerstone  effort  was  a  series  of 
traditional  and  distributed  testing-based  events  using  an  airborne  self-protection  jammer  (SPJ). 
The  SPJ  test  defined  a  simple,  repeatable  test  scenario.  The  scenario  was  executed  in  three 
traditional  test  environments  to  create  a  data  baseline.  The  test  scenario  was  then  executed  in 
two  ADS-enhanced  test  environments.  The  first  ADS-based  event  used  a  real-time  digital 
system  model  (DSM)  interacting  with  manned  threat  simulators  at  the  Air  Force  Electronic 
Warfare  Environment  Simulator  (AFEWES)  facility.  The  second  ADS-based  event  used  the  SPJ 
installed  on  an  F-16  suspended  in  the  anechoic  chamber  at  the  Navy’s  Air  Combat  Environment 
Test  and  Evaluation  Facility  (ACETEF).  The  data  from  all  tests  were  statistically  compared  in 
an  attempt  to  quantify  the  impacts  of  ADS. 
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ECM  =  electronic  countermeasures  HITL  =  hardware-in-the-loop  IADS  =  Integrated  Air  Defense  System 

RF  =radio  frequency  TSPI  =  time-space-position  information 

Figure  5.  Electronic  Warfare  SPJ  Test  Architecture 
The  other  efforts  used  by  JADS  to  examine  the  utility  of  ADS  were 

1)  the  OSD  CROSSBOW  Committee-sponsored  Threat  Simulator  Linking  Activity  (TSLA) 
effort, 

2)  the  DMSO-sponsored  High  Level  Architecture  Engineering  Protofederation  (EPF)  effort,  and 

3)  the  Army’s  Advanced  Distributed  Electronic  Warfare  System  (ADEWS)  development  effort. 

Each  of  these  efforts  added  to  the  SPJ  test  experience  to  provide  a  broader  understanding  of  the 
utility  of  ADS  to  EW  T&E. 

1. 4.3.1  EW  Test  Results  and  Conclusions 

Within  the  confines  of  the  SPJ  test  data,  JADS  concluded  that  ADS  architectures  that  allow  the 
capabilities  of  geographically  separated  facilities  to  be  combined  to  create  a  realistic  test 
environment  for  EW  devices  can  be  designed.  This  allows  the  same  test  environment  to  be  used 
for  SUT  representations  ranging  from  DSMs  to  operational  equipment.  Testing  in  a  common 
ADS-based  environment  represents  a  significant  departure  from  the  traditional  sequential  EW 
test  process. 

-  ADS  testing  architectures  requires  a  close  team  comprised  of  several  technical  experts 
spanning  several  disciplines  directed  by  a  system  integrator. 

-  The  architecture  produced  valid  results  for  both  the  DSM  and  actual  jammer  hardware. 


19 


-  Latency  within  the  closed-loop  interaction  was  aggressively  managed,  and  JADS  was  able  to 
meet  its  objective  for  more  than  95  percent  of  the  runs. 

-  HLA  appears  to  be  a  feasible  method  for  linking  simulations  for  T&E.  It  is  appropriate  to 
use  HLA  to  link  to  other  HLA-compliant  simulations/simulators,  but  the  T&E  community 
should  not  view  it  as  the  only  architecture  to  consider  in  designing  ADS  tests. 

-  Two  of  the  eleven  EW  testing  facilities  surveyed  in  1996  as  part  of  the  TSLA  effort  that  were 
appropriate  for  ADS-based  events  have  been  closed.  This  is  a  significant  erosion  in  the 
infrastructure  needed  to  design  and  execute  ADS  tests  limiting  the  traditional  EW  testing 
process. 

1.4.3. 2  Observations  for  EW  T&E 

JADS  assessment,  based  on  the  different  EW  Test  efforts,  is  that  ADS  has  varying  levels  of 
utility  for  EW  T&E.  These  levels  of  utility  depend  on  the  nature  of  the  EW  device  being  tested 
and  the  availability  of  suitable  test  facilities.  Single  function  EW  devices  and  federated  EW 
systems  are  expected  to  benefit  least  from  a  ADS-enhanced  process.  Only  radio  frequency  (RF) 
jammers  may  see  sufficient  benefit  to  outweigh  the  additional  cost  of  a  distributed  testing- 
enhanced  test  process.  Integrated  EW  systems  may  see  significant  benefits  where  a  single  test 
facility  is  not  capable  of  providing  all  the  stimulation  (including  the  closed-loop  SUT  versus 
manned  threat  interaction  for  systems  that  include  RF  jammers)  needed  to  simultaneously  test  all 
the  particular  integrated  EW  system  functions.  Systems  of  systems  testing,  such  as  that  required 
for  electronic  support  (ES)  systems,  should  see  significant  benefits  in  ADS-based  events. 

1.5  Other  ADS 

In  addition  to  the  three  specific  tests,  JADS  participated  to  varying  degrees  with  other  test 
agencies  and  organizations  using  ADS  technologies.  The  purpose  of  this  effort  was  to  leverage 
off  these  other  activities  to  broaden  our  conclusions  concerning  the  utility  of  ADS  across  the 
entire  spectrum  and  to  narrow  our  focus.  JADS  level  of  involvement  with  other  ADS 
organizations  ranged  from  passive  monitoring  to  full-scale  support  of  test  activities.  Where 
appropriate,  JADS  mapped  conclusions  and  lessons  learned  from  various  organizations  onto 
JADS  measures  and  objectives.  A  discussion  of  these  conclusions  and  lessons  learned  is 
included  under  the  relevant  measures  in  Section  2.  Table  2  lists  the  activities  and  programs 
JADS  was  involved  with  and  characterizes  our  degree  of  involvement. 
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Table  2.  Other  ADS  Activities  and  JADS  Degree  of  Involvement 


Activity 

Description 

JADS 

Involvement 

Common  Ground  Station 
(CGS)  Follow-On 

Operational  Test  and 
Evaluation  (FOT&E) 

Follow-on  test  and  evaluation  of  the 

Army’s  CGS 

Extensive 

VSTARS  Mission  Crew 
Training  System  (MCTS) 
Prototype 

Evaluation  of  VSTARS  MCTS  prototype  to 
support  and  contribute  to  the  overall 
mission  crew  training  and  continuation 
training  processes 

Extensive 

Synthetic  Environment 
Tactical  Integration  Virtual 
Torpedo  Project  (SETI-VTP) 

Test  of  the  real-time  interaction  of 
submarines  operating  on  range  with  high- 
fidelity  hardware-in-the-loop  (HITL) 
torpedo  simulations 

Moderate 

Project  Constellation 

Cooperative  effort  among  Army  test 
centers  to  link  their  facilities.  The  end 
state  for  Project  Constellation  is  a  virtual 
proving  ground  (VPG)  capability 

Extensive 

Bradley  Synthetic 

Environment  Operational 

Test  and  Evaluation  (SEOT) 

Program  to  perform  a  series  of  test-like 
events  in  a  synthetic  environment  (SE)  in 
parallel  with  corresponding  tests  in  a  live 
environment  to  evaluate  those  aspects  of 
the  SE  that  are  mature  enough  to  support 
future  testing  and  to  identify  and  quantify 
areas  of  the  SE  requiring  further 
development 

Moderate 

Threat  Simulator  Linking 
Activities  Study  (TSLA) 

CROSSBOW-sponsored  activity  to  provide 
the  facility  and  network  features  required 
to  support  the  T&E  of  electronic  warfare 
systems  where  the  test  environment  is 
composed  of  distributed  assets 

Extensive 

Joint  Strike  Fighter  (JSF) 

Employment  of  simulated 
aircraft/capabilities  in  a  simulated  theater 
of  operations  for  analysis  of  capabilities 

Limited 

Joint  Theater  Missile 

Defense  (JTMD) 

JT&E  to  investigate  and  evaluate  the 
capability  of  US  forces  to  conduct  TMD 
attack  operations  employing  existing 
systems 

Limited 
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Activity 

Description 

JADS 

Involvement 

Joint  Combat  Search  and 
Rescue  Joint  Test  and 
Evaluation  (JCSAR  JT&E) 

Linked  hardware-in-the-loop  simulators 
and  computer-generated  forces  at  Theater 

Air  Command  and  Control  Simulation 
Facility  (TACCSF)  and  Aviation  Test  Bed 
(AVTB)  simulation  facilities  to  execute 
JCSAR  events  for  a  rescue  force  launch 
order  to  recover  and  extract  isolated 
personnel 

Moderate 

Simulation,  Testing  and 
Operations  Rehearsal  Model 
(STORM) 

Test  support  tool  for  Force  XXI  Battle 
Command,  Brigade  and  Below  (FBCB^) 
tests  providing  a  combined  synthetic  and 
live  test  environment  that  replicates  C4I 
systems  and  information  flows  for  brigade- 
level  units  and  below 

Moderate 

Combat  Synthetic  Test  and 
Training  Range  (STTAR) 

Early  Joint  STARS  simulation  that  used 
Janus  to  display  mixed  virtual  and 
instrumented  real  forces 

Limited 

Joint  Electronic  Combat 

Test  Using  Simulation 
(JECSIM) 

Test  of  semi-active  missile  systems  against 
multiple  electronic  countermeasures 
(ECM)  techniques  employed  by  missile, 
fighter  and  bomber-sized  targets,  compare 
results  to  model  and  simulation  (M&S) 
predictions,  and  correlate  between  model 
results  and  live  test  results 

Limited 

Anti- Armor  Advanced 
Technology  Demonstration 
(A^ATD) 

Demonstration  to  develop  a  distributed 
synthetic  environment  capability  to  support 
anti-armor  weapon  system  virtual 
prototyping,  concept  formulation, 
requirements  definition,  effectiveness 
evaluation  and  mission  analysis  on  a 
combined  arms  battlefield  at  the  battalion 
task  force  and  brigade  level 

Limited 

Advanced  Distributed 
Electronic  Warfare  System 
(ADEWS) 

Simulation  of  the  effects  of  the  R330 
januner  on  friendly  communication 
equipment  including  the  Single-Channel 
Ground  and  Airborne  Radio  System 
(SINCGARS)  radio 

Extensive 
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2.0  Analysis  of  Test  Objectives 
l.lUtility 


To  determine  the  utility  of  ADS  to  T&E,  JADS  assessed  the  validity  of  data  from  tests  using 
ADS,  as  well  as  the  cost  effectiveness  and  other  benefits  of  using  ADS.  ADS  has  utility  if 
performance  data  compare  favorably  to  reference  data  and  if  ADS  data  are  timely,  accurate, 
reliable,  correct  and  otherwise  represent  real-world  systems  data.  The  first  step  in  the  analysis 
process  was  to  determine  if  and,  where  possible,  to  what  degree  ADS  provides  valid  data.  The 
first  objective  of  the  first  JADS  issue  addressed  data  validity.  The  second  objective  related  to  the 
cost  effectiveness  and  benefits  of  using  distributed  testing. 


Issue 

Objectives 

Issue  1:  What  is  the  present  utility  of  ADS, 
including  DIS,  for  T&E? 

Objective  1-1:  Assess  the  validity  of  data 
from  tests  using  ADS,  including  DIS, 
during  test  execution 

Objective  1-2:  Assess  the  benefits  of  using 
ADS,  including  DIS,  in  T&E 

2.1.1  Data  Validity  from  Tests  Utilizing  ADS 

To  determine  data  validity  the  JADS  a  priori  approach  was  to  take  a  previously  executed  live 
test,  replicate  that  test  in  an  ADS  environment  and  compare  the  results.  The  assumption  was  that 
live  data  were  by  definition  valid  or  “truth”.  This  approach  proved  problematic  and  more 
complicated  than  anticipated.  Live  data  are  often  incomplete,  either  because  of  missing 
environmental  information  or  inadequate  instrumentation.  Live  data  are  also  subject  to 
measurement  errors  that  may  or  may  not  be  well  documented.  The  fact  that  live  test  data  are 
generally  only  available  in  a  very  small  number  of  scenarios  is  another  limiting  factor  in  a 
comparison  approach  to  data  validation.  Because  of  the  uncertainties  and  complications 
associated  with  live  data,  it  was  not  always  obvious  that  live  data  rather  than  lab  data  should  be 
the  assumed  “truth.”  Because  of  these  uncertainties  and  complications  associated  with  using  live 
data  as  baseline  truth  JADS  used  additional  means  of  determining  ADS  data  validity. 

In  addition  to,  or  sometimes  in  lieu  of,  comparing  an  ADS  test  to  a  previously  executed  live  test, 
JADS  used  other  analysis  techniques  to  determine  data  validity.  These  techniques  included 
manual  and  automated  consistency  checking,  analysis  of  summary  statistics,  outlier  analysis, 
subject  matter  expert  (SME)  analysis  and  measure  of  effectiveness  (MOE)/measure  of 
performance  (MOP)  analysis.  These  techniques  either  supplemented  or,  in  some  instances, 
replaced  ADS  data  comparisons  to  live  data  to  determine  ADS  data  validity.  The  specific 
techniques  to  determine  data  validity  used  by  each  of  the  three  JADS  tests  are  discussed  in  detail 
in  the  individual  test  phase  and  final  reports.  In  all  cases  ADS  data  were  valid. 
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2.1.2  Benefits  of  Using  ADS  for  T&E 

The  structure  of  an  optimal  testing  program  should  be  based  on  the  appropriate  balance  between 
cost  savings  and  benefits.  To  determine  the  benefits  of  any  new  capability  or  technology  there 
must  be  objective  standards  against  which  the  performance  of  the  new  capability  or  technology  is 
measured.  These  performance  measures  or  standards  can  be  both  quantitative  and  qualitative. 
This  report  categorizes  performance  or  effectiveness  measures  as  either  standards  related  to 
enhanced  testing  capability  (is  it  better  or  faster?)  or  to  standards  related  to  cost  reduction 
potential  (is  it  cheaper?). 

2.1.2.1  Enhanced  Test  Capability 

JADS  demonstrated  that  ADS  could  overcome  many  of  the  traditional  limitations  and  problems 
with  test  and  evaluation.  ADS  allows  a  richer  more  reactive  environment  to  be  created  earlier  in 
system  development.  Traditional  single  model  or  model-versus-model  analysis  is  not  as  reactive 
as  simulations  where  human  intelligence  is  allowed  to  affect  appropriate  system  actions.  Human 
operators  are  an  integral  part  of  many  weapons  systems  and  need  to  be  part  of  early  system 
testing  if  possible.  For  example,  using  models  to  determine  jammer  effectiveness  against 
manned  threats  ignores  the  human  operator’s  ability  to  recognize  the  target  in  the  jammed 
display  increasing  the  risk  that  the  jammer  will  be  ineffective.  By  allowing  the  digital  model  to 
interact  with  the  manned  threat  simulators,  ADS  allows  the  system  developer  to  reduce  the 
development  risk  by  measuring  jammer  effectiveness  early  on.  In  the  PGM  example,  linked 
laboratories  can  provide  reproducible,  higher  confidence  results.  Missile  testing  using  a  linked 
laboratory  distributed  testing  architecture  is  more  reproducible  than  live  testing,  because  scenario 
conditions  are  more  readily  controlled  and  trials  can  be  replayed  for  additional  PGM  responses. 
This  allows  more  trials  to  be  combined  for  analysis,  giving  greater  confidence  in  evaluation 
results.  ADS  also  injects  more  realism  than  analytical  models  since  actual  hardware  is  used  and 
linked  simulation  is  often  more  realistic  than  stand-alone  HWIL  laboratories.  ADS  allows  the 
test  designer  to  take  advantage  of  the  laboratories  inherent  abilities  to  provide  secure  evaluation 
of  classified  electronic  countermeasures  (ECM)  techniques  and  increase  force  density  or 
representation  through  the  use  of  simulation.  In  the  C4ISR  arena,  ADS  allows  the  force  density 
of  the  scenario  to  be  increased  affordably.  The  number  of  friendly  and  threat  systems  can  be 
increased  by  representing  them  with  either  manned  laboratories  (if  realistic  man-in-the-loop 
control  of  the  systems  is  needed)  or  DSMs  (if  scripted  behavior  is  acceptable).  The  inability  to 
evaluate  system  performance  in  combat-representative  environments  is  a  common  limitation  in 
OT&E  and  an  area  in  which  ADS  can  improve  the  operational  test  (OT)  environment.  The 
ability  of  ADS  to  create  affordable,  large-scale  and  complex  environments  for  the  SUT  could 
mean  more  thorough  testing.  That,  in  turn,  could  provide  early  identification  of  problems  that 
might  otherwise  go  unnoticed.  There  is  also  a  potential  for  reducing  test  duration  by  using 
multiple  facilities  in  an  integrated  environment  rather  than  using  them  sequentially. 
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JADS  found  enhanced  test  capability  for  ADS  in  each  of  the  areas  investigated.  Key  areas  of 
utility  are  discussed  further  in  the  JADS  Special  Report  on  the  Cost  and  Benefits  of  Distributed 
Testing  which  can  be  found  at  www.jads.abq.com."^ 

2.1.2.2  Cost  Reducing  Potential 

Another  way  of  categorizing  the  benefits  of  using  ADS  relates  to  the  cost  reducing  potential. 
The  use  of  ADS  may  directly  result  in  cost  savings  and/or  reduce  overall  program  costs  by 
avoiding  costs. 

A  live  test  that  requires  multiple  platforms  be  brought  to  and  assembled  at  a  test  range  is  one 
example  of  the  type  of  test  that,  for  some  cases,  would  be  more  efficiently  and  effectively 
accomplished  with  ADS  testing.  This  example  could  at  least  help  support  the  argument  for  a 
reduction  in  the  number  of  expensive  live  tests  required.  Software  (SW)  developed  for  ADS 
may  be  reused  to  support  traditional  testing  methods.  One  example  of  this  is  from  the  JADS 
System  Integration  Test,  Live  Fly  Phase.  The  special  purpose  interface  developed  to  connect 
aircraft  to  the  Advanced  Medium  Range  Air-to-Air  Missile  (AMRAAM)  Ground  Lab  was  later 
used  for  troubleshooting  in  traditional  testing  methods.  VSTARS  is  probably  the  best  example 
of  SW  developed  for  ADS  with  reuse  capability.  However,  more  than  software  fits  in  this 
category.  For  example,  AFEWES  manned  simulators  were  used  for  decades  in  traditional  T&E. 
They  were  used  by  JADS  for  ADS-based  testing  by  adding  an  interface  to  the  facility.  Any 
potential  overlap  between  traditional  and  ADS  methods  should  be  addressed  by  feasibility 
analysis. 

The  empirical  data  from  the  JADS  tests  have  provided  the  JTF,  already  experienced  in 
conventional  T&E  techniques,  with  sufficient  confidence  to  identify  benefits  and  savings  arising 
from  the  use  of  ADS.  Several  uses  for  ADS  testing  technologies  have  been  identified  that 
support  cost  savings  for  traditional  tests  and  across  the  program  that  should  be  considered  when 
assessing  the  cost  of  ADS  to  the  overall  program. 

-  ADS  test  analysis  results  can  indicate  where  live  testing  can  best  be  focused  and  may  reduce 
the  need  for  some  live  tests. 

-  ADS  results  in  a  synthetic  environment  (SE)  that  can  support  other  areas  of  the  program, 
other  programs,  and  other  DoD  initiatives.  For  example: 

-  The  SE  capability  supports  system  design  and  development,  training  simulation,  and 
training  for  the  live  test  community. 

—  It  can  be  used  for  early  operational  assessments,  development  of  tactics,  techniques  and 
procedures  before  system  testing,  test  rehearsal,  verification  of  data  sources  and  data 
reduction  techniques,  and  to  determine  whether  adequate  data  are  collected  to  evaluate 
test  measures. 


■*  After  1  March  2001,  refer  requests  to  the  Joint  Program  Office  Technical  Library,  2001  North 
Beauregard  St.,  Suite  800,  Alexandria,  Virginia  22311. 
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—  In  some  cases,  other  test  programs  that  require  similar  input  can  reuse  an  SE  (e.g.,  the 
EXE  Test  SE,  with  minor  upgrades,  could  be  reused  for  the  Block  2  Army  Tactical 
Missile  System  (ATACMS)  OT&E). 

The  SBA  and  STEP  initiatives  will  advance  more  quickly  as  programs  initiate  development  of 
SEs  as  a  normal  step  in  the  program’s  life  cycle.  The  focus  of  SBA  is  reduced  cycle  time  —  the 
ability  to  use  the  technology  to  develop  systems  more  rapidly,  reducing  the  current  major  system 
cycle  time  from  15-20  years  to  7-10  years  (50  percent).  Currently,  most  systems  progress 
through  a  series  of  sequential  tests  at  multiple  facilities  using  the  unique  capabilities  of  the 
individual  facilities  to  address  different  test  objectives.  Distributed  testing  using  ADS 
technologies  can  be  used  to  link  the  individual  facilities  and  conduct  concurrent  testing  of 
multiple  test  objectives,  thus  providing  the  opportunity  for  significant  time  and  cost  savings.  If 
shortening  the  acquisition  timeline  results  in  cost  savings,  then  there  is  a  powerful  argument  for 
using  the  technology.  Major  test  programs  often  employ  a  variety  of  tools  including  physical 
models,  force-on-force  models,  hardware-in-the-loop  (HITL)  facilities,  open  air  testing,  system 
integration  labs,  simulators,  measurement  facilities  and  the  like.  In  some  cases,  testers  take 
advantage  of  large-scale  field  exercises  (which  are  becoming  increasingly  rare).  In  many  cases, 
the  SUT  must  be  moved  from  facility  to  facility  in  a  sequential  stream.  Because  of  scheduling 
issues  with  high  use  facilities,  there  is  often  a  significant  amount  of  slack  in  test  program  time 
lines.  Where  large-scale  field  exercises  are  the  test  vehicle  of  choice,  there  may  be  a  year  or 
more  between  test  opportunities.  Additionally,  the  use  of  ADS  can  reduce  total  testing  costs  by 
replacing  a  limited  number  of  live  tests  with  ADS-based  events. 

Cost  avoidance  is  the  notion  that  ADS  can  help  perform  more  complete  testing  earlier  in  a 
program,  identifying  failure  modes  and  other  problems  earlier  when  they  can  be  fixed  cheaper 
and  faster  than  when  they  are  discovered  later  in  the  system’s  life  cycle.  For  example,  the  ability 
of  ADS  to  incorporate  man-in-the-loop  provides  an  opportunity  for  cost  avoidance  when  the 
technology  is  used  as  a  test  rehearsal  and  training  tool.  Test  participants  can  climb  the  early  part 
of  the  learning  curve  in  the  virtual  environment,  and  the  test  schemes  can  be  refined  prior  to  the 
use  of  very  expensive  test  facilities.  The  time  associated  with  lost  test  events  could  therefore  be 
reduced.  In  some  situations,  ADS  can  reduce  the  risk  and  cost  of  wasted  live  fire  attempts  by 
providing  a  realistic  test  rehearsal  capability.  More  thorough  testing  should  result  in 
identification  of  system  deficiencies  earlier  in  the  life  cycle  when  they  can  be  fixed  more 
efficiently  saving  schedule  and  dollars.  ADS  can  redo  live  tests  that  have  encountered  problems 
more  quickly  than  live  retesting  which  would  result  in  schedule  slips.  Traditional  tests  are  still 
required,  but  by  reducing  the  number  of  lost  test  events  one  could  save  money  overall. 

Testing  could  be  more  thorough,  complete  more  test  scenarios,  and  used  for  cases  where  live 
tests  are  limited  by  test  range  constraints,  weather,  equipment,  or  when  the  test  environment 
needs  to  be  very  complex.  For  example,  the  Joint  STARS  Multiservice  Operational  Test  and 
Evaluation  (MOT&E)  was  originally  planned  to  be  conducted  over  the  National  Training  Center 
at  Fort  Irwin,  California,  with  300-500  military  vehicles  in  the  maneuver  area.  Background 
traffic  from  Los  Angeles,  California,  and  Interstate  10  would  have  provided  a  more  operational 
load  on  the  system,  however,  would  not  have  provided  an  increased  load  on  the  operators.  This 
approach  was  taken  because  the  test  planners  realized  they  would  never  be  able  to  provide  an 
operationally  representative  test  environment  short  of  an  actual  war.  The  JADS  ETE  Test  was 
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able  to  represent  10,000  vehicles  arranged  and  maneuvering  in  a  doctrinally  correct  threat 
laydown  of  enemy  corps.  This  provided  a  much  more  representative  operational  environment  to 
perform  OT  and  provided  the  testers  with  a  repeatable  environment  where  ground  truth  was 
known. 

2.1.2.3  Capability  to  Support  Early  Stages  of  the  Acquisition  Process 

The  DoD  is  seeking  to  streamline  the  acquisition  process  by  the  use  of  simulation  technology 
through  a  strategy  called  Simulation  Based  Acquisition.  The  goals  of  SBA  are  to 

-  substantially  reduce  the  time,  resources,  and  risk  associated  with  the  entire  acquisition 
process, 

-  increase  the  quality,  military  worth,  and  supportability  of  fielded  systems  while  reducing 
total  ownership  costs  throughout  the  total  life  cycle,  and 

-  enable  integrated  product  and  process  development  across  the  entire  acquisition  life  cycle. 

A  key  element  of  SBA  is  the  STEP,  which  is  defined  as  “an  iterative  process  that  integrates 
simulation  and  test  for  the  purpose  of  interactively  evaluating  and  improving  the  design, 
performance,  joint  military  worth,  survivability,  suitability,  and  effectiveness  of  systems  to  be 
acquired  and  improving  how  those  systems  are  used.” 

Under  the  SBA  and  STEP  concepts,  the  cost  of  testing  is  reduced  because  investments  in 
simulations  in  early  acquisition  phases  can  be  leveraged  rather  than  duplicated  in  later  stages. 
Reliability  is  increased  because  of  the  iterative  approach  inherent  to  SBA  and  STEP  where 
results  from  field  tests  are  incorporated  back  into  models  in  a  model-test-model  paradigm. 
Overall  cost  of  acquisition  is  reduced  because  system  evaluators  can  merge  information  from 
multiple  acquisition  phases  maximizing  insight  to  system  performance. 

ADS  is  particularly  useful  in  supporting  SBA.  ADS  has  relevance  in  these  areas.^ 

-  ADS  can  provide  a  framework  for  the  integration  of  models  developed  by  different 
acquisition  programs,  since  the  use  of  ADS  techniques  (e.g.,  linking  via  network  interfaces 
and  data  protocols)  can  permit  disparate  simulations  to  be  linked,  as  demonstrated  during 
JADS  testing. 

-  ADS  can  expedite  the  association  of  results  from  live  tests  of  a  C4ISR  element  with  the 
output  of  simulations  of  the  element.  Using  the  same  approach  as  the  ETE  Test,  the  element 
can  either  be  represented  by  the  actual  hardware  or  by  a  simulation  and  linked  in  either  case 
to  the  same  representations  of  the  other  elements  of  the  C4ISR  system.  Since  the  same 
scenarios  and  synthetic  environment  can  be  used,  correlation  of  results  between  the  live 
element  test  and  the  element  simulation  is  relatively  straightforward  and  can  support  the 
model-test-model  process  at  the  element  level. 


^  The  assessments  in  this  section  are  extrapolations  of  the  results  of  JADS  testing  and  related 
distributed  testing  programs  and  are  based  on  JADS  experience  and  insight. 
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-  ADS  can  also  support  the  model-test-model  process  at  the  system  of  systems  level  by 
providing  more  realistic  mission-level  test  results  that  can  be  used  to  refine  digital  system 
models  for  the  entire  C4ISR  system. 

-  ADS  supports  the  execution  of  the  Joint  Vision  2010  paradigm  that  requires  realistic 
battlespace  environments  populated  with  many  weapon  systems  and  threats.  In  particular, 
ADS  allows  the  large-scale,  complex  environment  evaluations  needed  for  C4ISR  systems. 

-  ADS  enables  the  linking  and  integration  of  geographically  distributed  resources  from 
different  system  representation  domains  (e.g.,  DSM,  HITL  lab,  ISTF,  battle  labs,  open  air 
range  [OAR])  that  can  lower  test  costs. 

-  ADS  supports  experimentation  of  emerging  war  fighting  concepts  and  testing  new  weapon 
systems. 

Further,  ADS  techniques  can  be  applied  to  all  C4ISR  element  acquisition  phases. 

-  Concept  Exploration.  If  a  C4ISR  element  DSM  becomes  available  during  this  initial  system 
acquisition  phase,  ADS  linking  techniques  can  be  used  to  provide  a  more  realistic  battlefield 
environment  and  to  permit  human  interactions  with  the  simulated  element.  This  capability  is 
especially  useful  for  development  of  a  concept  of  operation  (CONOPS)  for  the  emerging 
element  system. 

-  Program  Definition  and  Risk  Reduction.  As  prototype  C4ISR  element  system  hardware  is 
developed,  ADS-based  testing  can  be  expanded  and  refined.  ADS  configurations  can  be 
used  to  support  early  operational  assessments  and  for  more  realistic  specification  compliance 
testing  during  DT&E. 

-  Engineering  and  Manufacturing  Development.  Mission-level  evaluations  have  replaced 
traditional  requirements-based  OT&E  to  determine  whether  the  system  supports  the 
warfighter  and  ADS-based  testing  is  well  suited  for  this  application.  By  using  ADS,  the 
C4ISR  system  can  be  placed  in  a  realistic  operational  environment  and  valid  data  can  be 
collected  for  the  evaluation  of  operational  measures. 

-  Production,  Fielding/Deployment  and  Operational  Support.  By  permitting  operator-in-the- 
loop  operations  with  tactical  hardware,  ADS  can  support  the  development  of  tactics  and 
operational  procedures  in  conjunction  with  realistic  training.  ADS  can  also  be  used  for  the 
development  of  integrated  logistics  concepts. 

ADS  is  also  useful  for  supporting  the  iterative  spiral  development  process.  As  the  C4ISR 

element  undergoes  improvements  and  upgrades,  ADS  can  be  used  to  more  realistically 

-  verify  the  element  system  design, 

-  confirm  that  design  risks  have  been  controlled, 

-  certify  readiness  for  operational  testing,  and 

-  evaluate  the  system’s  operational  effectiveness,  suitability,  and  survivability. 
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2.1. 2.4  Capability  to  Support  T&E  Planning  and  Test  Rehearsal 

Although  ADS  technology,  in  and  of  itself,  does  not  help  to  support  traditional  test  planning  and 
rehearsal,  incorporating  aspects  of  ADS  into  tests  can  impact  planning  and  rehearsal  in  several 
areas  including  facilitating  test  concept  and  design,  tactics  development,  and  test  preparation. 
ADS  possesses  great  potential  for  making  improvements  in  these  areas;  yet,  there  is  sometimes  a 
price  to  be  paid  for  the  added  flexibility  and  complexity  that  accompanies  these  improvements. 
The  following  paragraphs  highlight  both  the  additional  value  and  complexity  involved  in  T&E 
planning  and  test  rehearsal  using  ADS,  as  experienced  during  the  three  JADS  test  efforts. 

ADS  offers  great  potential  for  improving  test  concept  and  test  design,  as  demonstrated  by  the 
impact  of  its  use  in  the  ETE  Test.  The  original  Joint  STARS  operational  test  plan  covered  only  a 
small  subset  of  its  capability.  With  the  creation  of  VSTARS,  a  virtual  model  of  Joint  STARS, 
and  the  inclusion  of  ADS  to  the  test  concept  and  test  design,  JADS  demonstrated  that  a  far  more 
complex  and  credible  test  could  be  planned,  rehearsed,  and  performed  to  test  the  Joint 
STARSWSTARS  capabilities.  Without  the  use  of  ADS,  operational  test  designs  for  systems  like 
Joint  STARS  may  remain  limited  in  scope  and  complexity. 

Similarly,  the  JADS  EW  Test  showed  that  flexibility  and  complexity  could  be  easily  added  to  a 
test  concept  and  design  by  utilizing  ADS  technology.  Just  prior  to  executing  Phase  3,  an 
additional  simulated  threat  capability  belonging  to  one  of  the  linked  facilities  was  added  to 
address  the  impacts  of  system  loading  on  both  network  and  SUT  capabilities.  The  HTTL  threats 
at  one  facility  and  the  simulated  threats  from  another  were  merged  into  a  common  environment 
to  provide  a  more  robust  test  for  the  SUT.  The  test  demonstrated  that  ADS,  in  allowing  for  the 
combination  of  test  resources  from  multiple  facilities  and  the  addition  of  assets  late  in  the  game, 
offers  flexibility  and  complexity  for  test  planning,  design,  and  execution  in  the  test  environment. 

The  SIT  LSP  was  another  example  of  flexibility  that  enhanced  test  planning.  ADS  allowed  the 
LSP  to  use  an  asset  that  included  a  man-in-the-loop  instead  of  a  constructive  simulation.  ADS 
linking  provided  for  the  use  of  the  F-14  simulator  at  Point  Mugu,  California,  operated  by  an 
operational  pilot.  The  addition  of  assets  that  more  closely  represented  real-world  conditions 
enhanced  test  execution.  With  the  increasing  complexity  and  interoperability  of  weapon 
systems,  it  is  less  likely  that  all  the  assets  required  for  a  robust  test  will  be  located  at  a  single  site. 
ADS  allows  test  planners  to  consider  test  designs  that  include  the  linking  of  assets  required  to 
support  quality  testing. 

Using  ADS  can  reduce  the  complexity  of  other  tasks  associated  with  testing,  such  as  data 
retrieval  and  reduction,  which,  in  turn,  reduce  the  amount  of  test  planning  and  rehearsal  required 
to  coordinate  such  efforts.  For  the  ETE  and  EW  tests,  data  retrieval  and  reduction  procedures  for 
ADS  phases  were  much  simpler  than  those  used  in  non-ADS  tests  or  test  phases.  The  ETE  Test 
data  retrieval  and  reduction  procedures  were  facilitated  by  the  dedicated  network  connections  at 
all  test  locations  that  allowed  for  easy  transfer  and  analysis  of  data  from  all  sites.  Similarly,  for 
the  EW  Test,  the  network  connections  to  the  participating  test  agencies  in  Phases  2  and  3 
allowed  data  to  be  transferred  more  simply  and  accurately  than  for  the  Phase  1  OAR  and  HITL 
tests.  EW  Test  data  reduction  was  also  simplified  because  each  distributed  testing-based  test 
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relied  on  just  one  data  file  for  collecting,  transferring,  and  manipulating  SUT  performance  data. 
The  OAR  and  HUL  test  phases  used  two  to  ten  separate  files  to  provide  the  same  information. 

In  some  cases,  adding  flexibility  and  complexity  to  test  concept  and  test  design  by  distributing 
the  test  also  adds  complexity  to  the  test  procedures  by  calling  for  more  coordination  among  test 
agencies.  For  the  ETE  Test,  coordination  among  different  test  agencies  became  the  largest 
obstacle  in  successful  test  execution.  For  the  EW  Test,  coordination  of  the  HLA  architecture  and 
configuration  management  were  more  difficult  to.  manage  than  non-ADS  tests.  The  SIT 
experienced  similar  issues  with  the  added  complexity  of  coordinating  test  design,  test  execution, 
and  test  control  among  agencies  spread  across  the  United  States. 

ADS  also  impacts  the  cost  of  test  concept  and  test  design.  The  price  of  added  complexity  and 
flexibility  is  no  small  consideration.  The  implementation  of  the  network  for  the  EW  Test  was 
the  single  most  impacting  task  in  preparing  for  the  tests.  Once  these  networks  are  established, 
however,  the  complexity  of  test  design  and  test  execution  can  be  greatly  decreased  and  costs 
reduced  for  subsequent  test  events.  The  ETE  Test  had  different  experiences  in  the 
implementation  of  the  ADS-based  events.  For  the  ETE  Test,  the  network  was  established,  but 
the  coordination  among  test  agencies  was  still  a  major  issue  in  successful  test  execution. 

The  Sir  experiences  were  similar  to  those  of  the  EW  Test.  The  integration  efforts  required  to 
establish  the  overall  network  were  extensive.  Once  the  integration  efforts  were  accomplished, 
ADS  allowed  for  the  refinement  of  the  test  execution  in  a  near  real-time  manner.  This  flexibility 
allowed  for  changes  to  be  made  quickly  resulting  in  the  test  objectives  being  met. 

The  complexities  facing  the  JADS  JTF  may  well  be  lessened  for  future  testers  who  use  ADS. 
Work  on  the  T&E  infrastructure  and  on  integration  approaches  such  as  HLA  may  ease  the 
difficulties  of  design  and  implementation. 

2.1.2.5  Capability  to  Overcome  T&E  Shortfalls 

During  the  feasibility  study  prior  to  the  JADS  JT&E,  a  survey  of  existing  conventional 
development  and  operational  test  limitations  was  conducted.  Three  hundred  and  sixty-one  total 
limitations  were  extracted  from  test  reports  and  T&E  master  plans.  Of  the  361  T&E  shortfalls 
identified  during  the  Joint  Feasibility  Study  (JFS),  it  was  determined  that  the  results  of  the  JTE 
could  be  extended  to  address  the  utility  of  using  ADS  T&E  methodologies  to  satisfy  276  of  those 
(76%).  Those  276  were  sorted  into  40  categories  as  shown  in  Table  3.  Table  3  rank  orders  the 
40  categories  according  to  the  number  of  times  a  given  shortfall  was  found  during  the  T&E 
requirements  survey. 
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Table  3.  T&E  Shortfalls  Addressed  by  JT&E 


SHORTFALL 

No.a 

METHOD^ 

Insufficient  test  articles 

25 

1,2 

Electronic  combat  not  allowed  or  limited/restricted 

21 

1 

Performance  restrictions 

20 

2 

Insufficient  battlespace 

19 

7,8 

Inadequate  instrumentation 

18 

1,2,5, 7, 8 

Inadequate  quantity  and  types  of  threat  systems 

17 

1 

Lack  of  systems  for  interoperability  testing 

16 

1 

Improper  threats  used 

14 

1,4 

Insufficient  number  of  test  events 

11 

2 

Ordnance/chaff/flare  restrictions 

10 

7 

Inadequate  quantity  and  types  of  targets/drones 

8 

1 

Lack  of  systems  for  compatibility  evaluation 

8 

1 

Human  interaction  not  represented  in  simulation 

8 

4 

Inadequate  quantity  and  types  of  friendly  systems 

7 

1 

Non-production  representative  test  assets 

7 

2 

Test  area  time/availability  restrictions 

7 

2 

Non-representative  force  levels 

6 

1 

Poor  fidelity  of  mod/sim  used  for  testing 

5 

1 

Constrained  concept  of  operations 

4 

2,8 

Inability  of  mod/sim  to  replicate  events/entities 

3 

1 

Multi-dimensional  threat  lacking 

3 

1 

Electromagnetic  environment  not  representative 

3 

1 

Poor  fidelity  of  emulators/simulators  used  as  targets/threats 

3 

1 

Incorrect  techniques/procedures  used 

3 

5 

Unrealistic  test  scenarios 

3 

5 

Data  collection  from  mod/sim  results  difficult 

3 

5 

Security  restrictions 

3 

7 

^No.:  Number  of  times  shortfall  was  found  during  T&E  requirements  survey 

‘^METHOD:  Applicable  ADS  T&E  Methodology; 

1:  Add  Assets 

2:  Increase  Test  Length,  Events,  Repetitions 

3:  Real-Time  Endgame  Analysis/Damage  Assessment 

4:  Human  Factors/Live  Response 

5:  Test  Planning 

6:  System  Development 

7:  Robustness  of  the  Physical  Environment 

8:  End-To-End  Testing  and/or  Post-Test  Evaluation 
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Table  3.  T&E  Shortfalls  Addressed  by  JT&E  (cont.) 


SHORTFALL 

Inadequate  number  of  simulation  entities 

2 

1 

Insufficient  test  resources  (supplies) 

2 

1 

Indirect  fire  tactics  and  capabilities  not  well  represented 

2 

1 

Joint/combined  operations  lacking  or  improperly  represented 

2 

1 

Real-time  instrumentation  lacking 

2 

1 

Target  control  lacking 

2 

1 

Real-time  analysis  lacking 

2 

3 

Incompatibility  of  collected  data 

2 

5 

Insufficient  personnel  resources 

1 

1 

Mod/sim  inflexible 

1 

1 

Environmental  impact  restrictions 

1 

1,7 

Improper  friend,  foe,  and  neutral  tactics 

1 

5 

Integrated  avionics  difficult  to  test 

1 

6 

TOTAL 

276 

^No.:  Number  of  times  shortfall  was  found  during  T&E  requirements  survey 

t'METHOD:  Applicable  ADS  T&E  Methodology: 

1 :  Add  Assets 

2:  Increase  Test  Length,  Events,  Repetitions 

3:  Real-Time  Endgame  Analysis/Damage  Assessment 

4:  Human  Factors/Live  Response 

5:  Test  Planning 

6:  System  Development 

7:  Robustness  of  the  Physical  Environment 

8:  End-To-End  Testing  and/or  Post-Test  Evaluation 

The  remaining  85  shortfalls  which  were  not  addressed  by  the  JT&E  are  listed  in  Table  4.  Of 
these: 

-  18  shortfalls  (NOTE  #1  in  Table  4.)  could  have  been  addressed  by  specialized 

application  of  ADS  T&E  methodologies,  but  were  not  addressed  because  of  scoping 
limitations. 


-  46  shortfalls  (NOTE  #2  in  Table  4.)  could  not  be  addressed  because  of  inadequate 

environmental  representations  in  state  of  the  art  models  and  simulations,  a  current 
technology  limitation. 

21  shortfalls  (NOTE  #3  in  Table  4.)  could  not  be  addressed  by  applications  of  ADS 
methodologies.  The  use  of  ADS  cannot  overcome  the  fact  that  some  VV&A  may  be 
difficult  and  costly;  however,  this  fact  doesn’t  prevent  the  application  of  ADS.  If 
costly  V&V  or  model  development  are  issues,  they  are  scoping  limitations  as  opposed 
to  application  limitations.  Additionally,  it  was  JADS  experience  that  the  costs  of 
model  development  and  VV&A  are  manageable. 
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Table  4.  T&E  Shortfalls  Not  Addressed  by  JT&E 


SHORTFALL 

No.a 

METHOD^ 

NOTEd 

Limited  training  in  use  of  test  articles 

14 

5 

1 

Inadequate  cluster  munitions  representation  in  models 

2 

1 

1 

Insufficient  logistical  support  for  test  assets 

2 

2 

1 

Non-representation  of  battlespace 

15 

7 

2 

Non-operational  environmental  test  conditions  " 

13 

7 

2 

Restrictions  on  laser/emp  use 

9 

7 

2 

Environmental  restrictions 

8 

7 

2 

Terrain  modification  not  allowed 

1 

7 

2 

VV&A  difficult  and  costly 

9 

None^ 

3 

Mod/sim  costly 

3 

None^ 

3 

Misc 

9 

None^ 

3 

TOTAL 

85 

^No.:  Number  of  times  shortfall  was  found  during  T&E  requirements  survey 
‘’METHOD:  Applicable  ADS  T&E  Methodology  (see  Table  3.) 

^None:  ADS  T&E  Methodologies  cannot  correct  these  shortfalls 
^NOTE:  See  text  for  explanation 

We  conclude  that  294  of  the  361  shortfalls  could  have  been  addressed  in  the  JT&E  with  current 
or  near-term  ADS  technology,  and  276  of  these  were  addressed  (94%  of  those  that  could  be 
addressed).  Hence  the  JADS  results  have  broad  applicability. 

The  40  shortfall  categories  listed  in  Table  3  map  to  JADS  Issue  1,  Objective  1-2,  Subobjective  1- 
2-3,  Assess  ADS  capability  to  support  T&E  execution.  Measures  1  through  40.  Table  5  identifies 
the  section(s)  of  the  final  report  in  which  these  40  measures  are  addressed. 
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Table  5.  Final  Report  Section  Addressing  T&E  Shortfall  Categories 


Final  Report  Section 

JADS  Measure 

2.1.2.5.1 

1 -2-3-1  Degree  to  which  ADS  can  add  assets  to  test  execution 

1 -2-3-4  Degree  to  which  ADS  overcomes  the  traditional  T&E 
shortfall  of  insufficient  battlespace 

1-2-3-15  Degree  to  which  ADS  can  provide  non-production 
representative  test  assets 

1-2-3-17  Degree  to  which  ADS  can  provide  for  more 
representative  force  levels 

1-2-3-19  Degree  to  which  ADS  can  improve  upon  a  constrained 
concept  of  operations 

1-2-3-25  Degree  to  which  ADS  can  insure  realistic  test  scenarios 

1-2-3-28  Degree  to  which  ADS  can  increase  the  number  of 
simulation  entities 

1-2-3-29  Degree  to  which  ADS  can  add  resources  (supplies) 

1-2-3-39  Degree  to  which  ADS  can  facilitate  the  proper  use  of 
friend,  foe,  and  neutral  tactics 

2.1.2.5.2 

1 -2-3-9  Degree  to  which  ADS  increases  the  number  of  test  events 

1-2-3-16  Degree  to  which  ADS  can  overcome  test  area  time/ 
availability  restrictions _ 

2.1. 2.5.3 

1-2-3-13  Degree  to  which  ADS  can  represent  human  interaction 
in  simulation 

1-2-3-36  Degree  to  which  ADS  can  increase  personnel  resources 

2.1. 2.5.4 

1 -2-3-2  Degree  to  which  ADS  allows  for  electronic  combat 

1 -2-3-3  Degree  to  which  ADS  overcomes  performance 
restrictions 

1-2-3-10  Degree  to  which  ADS  can  over  come 
ordinance/chaff/flare  restrictions 

1-2-3-22  Degree  to  which  ADS  can  provide  for  representative 
electromagnetic  environments 

1-2-3-27  Degree  to  which  ADS  can  overcome  traditional  testing 
constraints  imposed  because  of  security  restrictions 

1-2-3-38  Degree  to  which  ADS  can  overcome  environmental 
impact  restrictions 

2.1. 2.5.5 

1 -2-3-7  Degree  to  which  ADS  overcomes  the  lack  of  systems  for 
interoperability  testing  associated  with  traditional  T&E 

1-2-3-12  Degree  to  which  ADS  can  increase  the  number  of 
systems  for  compatibility  evaluation 

2.1.2.5.6 

1-2-3-24  Degree  to  which  ADS  can  insure  that  correct  techniques 
and  procedures  are  used 

1-2-3-33  Degree  to  which  ADS  can  provide  for  target  control 

1-2-3-39  Degree  to  which  ADS  can  facilitate  the  proper  use  of 
friend,  foe,  and  neutral  tactics 
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Table  5.  Final  Report  Section  Addressing  T&E  Shortfall  Categories  (cont.) 


Final  Report  Section 

JADS  Measure 

2.1.2.5.7 

1-2-3-30  Degree  to  which  ADS  can  represent  indirect  fire  tactics 
and  capabilities 

2.1. 2.5.8 

1 -2-3-6  Degree  to  which  ADS  overcomes  traditional  T&E 
shortfall  of  inadequate  quantity  and  types  of  threat  systems 

1 -2-3-8  Degree  to  which  ADS  overcomes  the  use  of  improper 
threats  used  in  traditional  T&E 

1 -2-3-1 1  Degree  to  which  ADS  overcomes  the  traditional  T&E 
shortfall  of  inadequate  quantity  and  types  of  targets/drones 

1-2-3-14  Degree  to  which  ADS  can  provide  adequate  quantities 
and  types  of  friendly  systems 

1-2-3-21  Degree  to  which  ADS  can  provide  multidimensional 
threats 

1-2-3-39  Degree  to  which  ADS  can  facilitate  the  proper  use  of 
friend,  foe,  and  neutral  tactics 

2.1. 2.5.9 

1-2-3-40  Degree  to  which  ADS  can  test  integrated  avionics 

2.1.2.5.10 

1-2-3-18  Degree  to  which  ADS  can  provide  for  improved 
mod/sim  used  for  testing 

1-2-3-20  Degree  to  which  ADS  can  overcome  mod/sim’ s 
inability  to  replicate  events/entities 

1-2-3-23  Degree  to  which  ADS  can  provide  high  fidelity 
emulators  and  simulators  used  as  targets  and/or  threats 

1-2-3-26  Degree  to  which  ADS  can  facilitate  mod/sim  data 
collection 

1-2-3-37  Degree  to  which  ADS  can  overcome  mod/sim 
inflexibility 

2.1.2.5.11 

1-2-3-31  Degree  to  which  ADS  can  represent  joint/combined 
operations  and  capabilities 

2.1.2.5.12,  also 

2.2.1  -2.2.2 

1-2-3-34  Degree  to  which  ADS  can  provide  for  real-time  analysis 

1-2-3-35  Degree  to  which  ADS  can  overcome  incompatibility  of 
collected  data 

2.1.2.5.13,  also 

1 -2-3-5  Degree  to  which  ADS  overcomes  traditional  T&E 

2.2. 1.1 _ shortfall  of  inadequate  instrumentation _ 

1-2-3-32  Degree  to  which  ADS  can  provide  for  real-time 
instrumentation 


2.1. 2.5.1  Ability  of  ADS  to  Increase  Assets,  Force  Levels,  Entities,  and  Battlespace 
This  section  addresses  the  following  JADS  measures: 

JADS  Measure  1-2-3-1  Degree  to  which  ADS  can  add  assets  to  test  execution 

JADS  Measure  1 -2-3-4  Degree  to  which  ADS  overcomes  the  traditional  T&E  shortfall  of 
insufficient  battlespace 

JADS  Measure  1-2-3-15  Degree  to  which  ADS  can  provide  for  non-production 
representative  test  assets 

JADS  Measure  1-2-3-17  Degree  to  which  ADS  can  provide  for  more  representative  force 
levels 

JADS  Measure  1-2-3-19  Degree  to  which  ADS  can  provide  improve  upon  a  constrained 
concept  of  operations 

JADS  Measure  1-2-3-25  Degree  to  which  ADS  can  insure  realistic  test  scenarios 

JADS  Measure  1-2-3-28  Degree  to  which  ADS  can  increase  the  number  of  simulation 
entities 

JADS  Measure  1-2-3-29  Degree  to  which  ADS  can  add  resources  (supplies) 

JADS  Measure  1-2-3-39  Degree  to  which  ADS  can  facilitate  the  proper  use  of  friend,  foe, 
and  neutral  tactics 

Conventional  T&E  is  commonly  limited  by  battlespace  restrictions  and  an  inadequate  number  of 
entities.  Typical  tests  are  often  conducted  in  conjunction  with  training  missions  in  order  to 
acquire  a  suitable  battlespace  and  a  large  number  of  assets.  If  testers  rely  on  training  missions, 
they  have  only  minimal  control  over  test  specifics  and  their  test  must  be  limited  to  the  training 
scenario  being  executed.  ADS  allows  testers  to  overcome  this  T&E  shortfall  by  providing  a 
affordable  larger  battlespace  with  a  mix  of  live  and  virtual  assets,  reducing  the  number  of 
personnel  and  equipment  needed  for  the  test. 

The  size  of  the  battlespace  and  number  of  entities  were  primary  issues  for  the  ETE  Test.  The 
ETE  Test  was  executed  using  nearly  10,000  Janus  entities  for  each  vignette  and  a  virtual 
battlespace  of  150  square  kilometers  (km^).  It  would  be  impractical  for  a  typical  test  to  be 
conducted  with  such  a  large  number  of  entities  and  battlespace  because  of  the  cost  and  logistics 
involved,  even  if  the  test  was  conducted  in  conjunction  with  a  training  mission. 

For  the  ETE  Test,  the  battlespace  and  large  number  of  entities  gave  the  test  team  the  ability  to 
closely  reflect  the  number  and  types  of  forces  expected  of  an  Army  corps  functioning  in  an 
operational  theater.  The  testers  were  also  able  to  control  specific  aspects  of  interest  in  the 
scenarios  and  to  expand  the  test  concept  and  design  as  desired.  The  largest  tests  conducted  by 


36 


the  Army,  comparable  to  the  scenarios  tested  by  the  ETE  Test  team,  are  accomplished  at  the 
National  Training  Center  (NTC).  These  tests  are  executed  in  conjunction  with  NTC  using  a 
maximum  battlespace  of  about  100  km^.  The  results  show  the  great  increase  in  battlespace 
exhibited  by  the  Phase  4  test  over  traditional  testing  where  an  additional  50  km"  were  added  to 
the  battlespace.  In  addition,  the  battlespace  of  150  km"  used  by  the  ETE  Test  team  is  far  from 
the  maximum  that  ADS  is  capable  of  supporting.  This  battlespace  was  picked  for  its 
applicability  to  the  scenarios  used  during  the  Phase  4  test,  but  could  be  greatly  expanded  if 
necessary.  In  addition,  using  ADS  technology,  even  larger  numbers  of  entities  and  more 
battlespace  could  have  been  added.  The  Phase  4  ETE  results  imply  the  following  benefits  of 
ADS-based  testing; 

-  An  ADS-enhanced  live  test  environment  using  validated  simulations  can  provide  more 
realistic  threats  and  force  levels  than  those  offered  by  conventional  tests,  i.e.,  threats/force 
levels  otherwise  unavailable  because  of  cost  restrictions,  unobtainable  threats,  etc. 

-  C4ISR  system  testers  can  tailor  the  simulation  entities  operating  in  the  ADS  environment  to 
closely  reflect  the  forces  expected  in  operational  theaters,  thus  further  increasing  the 
relevance  of  the  collected  test  data. 

-  ADS  allows  testers  to  have  more  control  over  the  specific  aspects  of  the  scenarios  of  interest 
and  to  expand  their  test  concept  and  design.  A  typical  constraint  to  test  concept  development 
is  the  number  and  types  of  units  readily  available  for  a  test.  For  example,  a  conventional  test 
may  require  the  use  of  a  battalion,  but  because  of  a  prior  commitment  or  cost  these  personnel 
may  not  be  available.  In  contrast,  ADS  allows  testers  to  create  a  virtual  battalion  and  to  test 
their  concepts  with  a  minimum  number  of  personnel  and  equipment. 

To  a  lesser  extent,  the  EW  Test  also  offered  the  ability  to  increase  the  number  of  threats  and 
targets  available.  For  the  EW  Test,  the  ADS  technology  made  it  possible  to  add  as  many 
simulated  threats  to  the  engagement  as  there  were  RF  generators  at  the  test  facility,  adding 
unmanned  threats  to  the  scenario  to  go  along  with  the  manned  threats  used  at  AFEWES. 
Similarly,  the  EW  Test  could  increase  the  number  of  targets  easier  than  in  a  traditional  test. 
Through  script  manipulation,  multiple  targets  could  be  added  to  test  the  reactions  of  the  normal 
threats  to  multiple  targets. 

The  size  of  the  battlespace,  and  the  number  of  entities  were  not  issues  for  the  SIT  because  of  the 
scenarios  being  simulated.  However,  though  the  SIT  used  only  a  few  entities  during  testing 
efforts,  the  architecture  would  have  allowed  for  the  addition  of  other  entities  if  they  had  been 
required. 

With  traditional  testing  it  is  often  difficult  and  expensive  to  acquire  all  the  support  resources  and 
supplies  to  conduct  a  robust  test.  Most  applications  of  ADS  supported  distributed  testing  can  be 
used  to  overcome  a  shortage  or  lack  of  test  resources  (supplies)  typical  of  most  forms  of 
traditional  testing.  EW  testing  on  the  other  hand,  unlike  other  forms  of  traditional  testing,  uses 
few,  if  any,  expendable  resources  such  as  supplies.  Therefore,  there  is  little  if  any  need  to  use 
ADS  in  EW  testing  to  overcome  a  shortage  of  supplies. 
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Although  ADS  can  be  used  to  overcome  a  shortage  of  supplies  there  are  test  resources  unique  to 
the  use  of  ADS.  The  test  resources  (supplies)  involved  in  the  setup  and  execution  of  the  ETE 
test  were  examined  to  determine  which  resources  would  not  have  been  required  for  a  traditional 
test..  In  addition,  the  amount  of  resources  (supplies)  required  for  the  test,  due  solely  to  ADS- 
specific  requirements,  was  documented.  ADS  tests,  unlike  traditional  tests,  require  resources 
(supplies)  related  to  the  network  and  distributed  nodes.  Additional  network  and  communication 
equipment  is  needed  to  ensure  network  reliability  and  good  communication  at  all  nodes.  In 
addition,  an  increased  amount  of  computer  equipment  is  required  at  the  distributed  nodes  to  run 
the  simulations  involved  in  ADS  tests.  Any  support  materials  required  at  the  distributed  nodes 
would  also  account  for  an  increase  in  test  resources  (supplies)  for  ADS  tests. 

2. 1.2. 5.2  Ability  of  ADS  to  Increase  the  Number  of  Test  Events  and  Test  Time 

This  section  addresses  the  following  JADS  measures; 

JADS  Measure  1-2-3-9  Degree  to  which  ADS  increases  the  number  of  test  events 

JADS  Measure  1-2-3-16  Degree  to  which  ADS  can  overcome  test  area  time/availability 
restrictions 

Using  ADS,  testers  can  conduct  more  test  events  and  test  for  longer  periods  of  time  than  with 
traditional  T&E  because  of  the  reduced  cost  and  logistics  required  and  the  capability  to  maintain 
continuous  and  consecutive  testing.  All  three  JADS  tests  took  advantage  of  these  ADS  testing 
capabilities  to  increase  test  events  and  test  time. 

For  example,  during  Phases  2  and  4  of  the  ETE  Test,  the  test  team  was  able  to  test  for  more  than 
30  hours  in  a  1-week  period.  This  amount  of  test  time  over  a  similar  test  period  would  be  nearly 
impossible  to  match  with  a  traditional  test  using  real  assets  because  of  the  cost  and  logistical 
restrictions.  Similarly,  the  EW  Test  team  was  able  to  perform  271  runs  during  the  two  weeks  of 
both  the  Phase  2  and  Phase  3  tests.  During  the  Phase  1  OAR  test,  the  team  was  limited  to  14.4 
hours  of  testing  over  eleven  months,  obtaining  only  136  runs.  Without  the  ADS  testing 
capability  to  support  a  continuous  and  consecutive  test  period,  these  results  would  not  have  been 
possible 

During  the  SIT  LSP,  two  operational  pilots  replicated  the  actual  engagement  of  a  previously 
executed  live  missile  shot.  The  ADS  design  allowed  for  the  resetting  of  the  engagement  in  just  a 
few  seconds.  If  this  test  had  been  executed  by  live  aircraft  it  would  have  taken  5-10  minutes  to 
return  the  aircraft  on  the  range  to  the  start  position.  In  addition,  fuel,  weather,  or  flight-time 
limitations  did  not  limit  the  aircraft  or  pilots  in  the  SIT  LSP.  During  the  SIT  LFP,  the  team 
obtained  14  valid  runs  in  a  2-hour  sortie  that  included  refueling.  Additionally,  the  ADS 
configuration  allowed  for  the  review  of  engagement  results  while  the  aircraft  were  returned  to 
the  start  positions.  This  near  real-time  data  review,  which  is  not  available  during  conventional 
range  operations,  allowed  adjustments  to  be  made  for  the  next  run.  Conventional  live  testing 
would  have  allowed  one  run  and  analyzed  the  data  after  the  aircraft  had  landed.  ADS  allowed 
the  Sir  to  maximize  the  number  of  test  runs  and  to  meet  test  objectives. 
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2. 1 .2.5.3  Ability  of  ADS  to  Overcome  a  Lack  of  Personnel 
This  section  addresses  the  following  JADS  Measures: 

JADS  Measure  1-2-3-13  Degree  to  which  ADS  can  represent  human  interaction  in 
simulation 

JADS  Measure  1-2-3-36  Degree  to  which  ADS  can  increase  personnel  resources 

A  common  shortfall  of  traditional  T&E  is  the  lack  of  manpower  available  to  conduct  test 
activities.  This  shortage  in  personnel  resources  can  result  in  a  reduced  and  sometimes 
insufficient  testing  capability.  ADS  has  the  potential  to  overcome  a  lack  of  personnel  resources 
by  replacing  personnel  with  computer  simulations  which  in  turn  interact  with  other  virtual,  live 
and  constructive  representations.  For  example,  the  ETE  Test  used  the  Tactical  Army  Fire 
Support  Model  (TAFSM)  simulation  as  a  replacement  for  the  Army  personnel  that  would  have 
been  needed  in  similar  non-ADS  tests.  Another  example  is  the  simulated  use  of  ground  vehicles. 
Live  testing  requires  an  operator  per  vehicle  as  well  as  observer  personnel.  When  all  vehicles 
are  synthetic  only  a  simulation  operator  is  required.  Additionally,  ADS  allows  you  link 
personnel  who  would  otherwise  be  unavailable.  During  the  ETE  test  live  GSM  operators  in 
garrison  at  Ft.  Hood  Texas  who  were  unavailable  for  deployment  as  well  as  FDC  and 
intelligence  personnel  were  linked  to  the  test  environment. 

The  Srr  and  EW  Test  experienced  the  need  for  additional  personnel  to  successfully  execute  their 
ADS-enhanced  test.  The  ADS-based  phases  of  the  EW  Test  used  from  three  to  nine  more  people 
than  similar  non-ADS  tests  to  accomplish  the  same  goal.  In  this  case,  additional  personnel  were 
needed  to  monitor  network  performance  and  real-time  visualization  tools  during  the  test 
execution,  to  observe  simulation  equipment,  to  gauge  test  operator  performance,  and  to  watch  for 
system  failures  in  the  ADS  architecture.  The  SIT  had  results  similar  to  the  EW  Test  in  that 
personnel  were  needed  to  monitor  system  performance  during  the  ADS  portion  of  the  test.  The 
use  of  the  TCAC  during  the  LSP  was  an  example  of  additional  personnel  requirements  resulting 
from  the  use  of  ADS  supported  distributed  testing.  During  the  LSP,  personnel  were  required  to 
conduct,  control,  and  monitor  the  test  from  Albuquerque.  With  ADS-enhanced  testing,  a  central 
node  where  the  test  can  be  controlled  is  needed.  This  node  must  provide  the  ability  to  monitor 
the  entire  network  during  the  test.  In  addition,  all  the  nodes  must  provide  personnel  to  interface 
with  the  central  node.  These  personnel  requirements,  however,  are  not  necessarily  unlike  the 
requirements  for  traditional  testing.  Personnel  for  test  control  and  monitoring  are  required  for 
traditional  testing,  as  are  people  to  coordinate  at  each  test  site.  The  requirement  for  additional 
ADS  test  control  personnel  is  related  to  network  instrumentation  and  monitoring  requirements 
that  are  not  required  for  some  traditional  tests.  In  the  EW  test,  the  requirement  for  observers  at 
the  manned  threat  locations  was  not  an  ADS  requirement  as  well.  JADS  chose  to  place 
observers  at  each  threat  location  to  monitor  operator  performance  and  control  for  operator 
induced  variability. 

Overall,  simulation  of  personnel  actions  is  possible  for  testing  various  systems,  but  the 
requirement  for  monitoring  personnel  at  each  of  the  distributed  sites  may  exceed  the  personnel 
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requirement  using  a  nondistributed  test  approach.  In  field  tests  involving  dispersed  sites,  the 
need  for  monitoring  personnel  is  essentially  the  same. 

2.1. 2. 5.4  Ability  of  ADS  to  Overcome  Traditional  T&E  Restrictions 

This  section  addresses  the  following  JADS  measures; 

JADS  Measure  1-2-3-2  Degree  to  vrhich  ADS  alloys  for  electronic  combat 

JADS  Measure  1-2-3-3  Degree  to  which  ADS  overcomes  performance  restrictions 

JADS  Measure  1-2-3-10  Degree  to  which  ADS  can  over  come  ordinance/chaff^flare 
restrictions 

JADS  Measure  1-2-3-22  Degree  to  which  ADS  can  provide  for  representative 
electromagnetic  environments 

JADS  Measure  1-2-3-27  Degree  to  which  ADS  can  overcome  traditional  testing  constraints 
imposed  because  of  security  restrictions 

JADS  Measure  1-2-3-38  Degree  to  which  ADS  can  overcome  environmental  impact 
restrictions 

Traditional  T&E  is  often  adversely  affected  by  restrictions  imposed  by  performance,  safety, 
environmental,  and  security  factors.  Unsafe  conditions,  restricted  tactics,  reduced  maneuvering 
speeds  and  locations,  the  lack  of  test  and  communication  assets,  and  electromagnetic  and 
environmental  restrictions  are  just  some  of  the  limitations  that  can  negatively  impact  the 
robustness  of  a  typical  test. 

ADS  can  be  helpful  in  overcoming  these  shortfalls.  For  example  the  SIT  LSP  used  the  capability 
to  investigate  tactics  using  pilots  in  aircraft  simulators  linked  to  a  simulated  missile  iii  a 
hardware  in  the  loop  facility.  An  ADS  configuration  such  as  this  can  allow  pilots  to  examine 
portions  of  the  envelope  too  dangerous  to  examine  on  the  open  range.  While  tactics 
development  was  not  specifically  the  focus  of  this  test,  the  SIT  could  have  been  expanded  to  test 
new  and  improved  tactics  normally  too  dangerous  to  implement  in  a  nonsimulation-based  test. 

For  the  SIT,  ADS  offered  the  capability  to  perform  multiple  flyouts  of  the  same  missile 
engagement;  a  missile  engagement  that  could  not  have  been  flown  live  because  of  safety 
restrictions.  ADS  provided  the  capability  to  perform  more  repeatable  and  robust  testing  before 
expending  live  missile  assets.  Additionally,  ADS  can  overcome  ordinance,  chaff  and  flare 
restrictions.  Testing  with  actual  chaff  and  flares  is  restricted  because  of  the  potential  of  an  errant 
live  missile.  A  key  feature  in  the  SIT  LSP  Mission  #2:  Parametric  Study  Mission  was  the  use  of 
counter  measures  involving  the  ejection  of  flares.  Baseline  validation  profile  V2  involved 
variable  flare  dispense  times  based  on  the  automatic  transmission  of  a  flare  PDU  from  the  WSIC. 
Although  JADS  did  not  specifically  simulate  the  use  of  chaff  in  the  SIT  (or  EW)  test,  the  TSLA 
study  describes  how  this  can  be  done.  The  SIT  LFP  ADS  architecture  also  demonstrated  the 
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potential  to  use  ECM  techniques.  ECM  techniques  could  be  used  against  the  missile  and  fire 
control  radar  of  the  shooter  aircraft  in  a  SIT  LFP  test  configuration  overcoming  the  range 
restrictions  of  traditional  live  testing. 

For  the  ETE  Test,  maneuvering  speeds,  usage  of  simulated  hostile  territory  landscapes,  and  the 
addition  of  thousands  of  trackable  entities  resolved  many  problems  of  traditional  T&E  testing. 
Because  of  the  resolution  of  these  problems,  the  ETE  Test  was  able  to  execute  a  corps-size  test 
with  up  to  10,000  entities  using  tactics  and  movements  that  would  have  been  restricted  in  a  non- 
ADS  based  test.  There  is  no  environmental  impact  from  virtual  vehicles  operating  in  a  synthetic 
test  environment.  In  terms  of  overcoming  ordinance  restrictions  in  the  ETE  test,  ADS  allowed 
the  firing  of  a  synthetic  ATACMS  at  synthetic  troops.  This  would  be  impossible  to  do  in  a 
traditional  test. 

For  EW  tests,  non-ADS  tests  do  not  allow  live  missile  flyouts  against  a  SUT  or  target  aircraft. 
Instead,  missile  models  are  used  to  allow  OT&E  measurements  to  be  made  on  both  SUT  and 
threat  performance.  The  JADS  EW  test  was  designed  to  test  the  ability  of  ADS  to  overcome 
traditional  test  and  evaluation  shortcomings  related  to  the  use  of  electronic  combat.  The  results 
of  the  EW  test  demonstrate  that  ADS  can  overcome  many  of  the  T&E  shortfalls  associated  with 
electronic  combat  testing.  The  EW  test  RF  federate  was  designed  to  provide  for  a  representative 
electromagnetic  environment.  Unfortunately  JADS  was  unable  to  capture  a  representative 
environment  from  the  OAR  phase  of  testing,  but  conceptually  this  is  possible.  The  EW  test  did, 
however,  demonstrate  the  capability  to  use  jamming  techniques  that  the  Federal  Communications 
Commission  (FCC)  and  the  Federal  Aviation  Administration  (FAA)  would  not  allow  on  the  open 
range.  Although,  the  JADS  EW  Test  was  not  specifically  designed  to  address  possible 
environmental  issues  or  the  use  of  dangerous  maneuvers,  future  tests  could  accomplish  this  if 
HITL  aircraft  simulations  were  used.  The  EW  Test  used  a  scripted  aircraft  position  to  assess 
ADS  effects,  but  a  test  could  be  designed  that  allowed  pilot  interactions  with  threat  operators  to 
determine  a  more  realistic  outcome  for  an  OT&E  test. 

By  providing  large  numbers  of  assets,  depicting  unsafe  conditions  in  convoy  movements  and 
aircraft  and  missile  engagements,  and  applying  tactics  normally  restricted  in  typical  T&E,  the 
ADS  testing  technology  increased  test  robustness  for  the  three  JADS  tests.  In  all  cases  however, 
the  elimination  of  T&E  shortfalls  was  accomplished  at  the  cost  of  the  implementation  of  ADS, 
which  can  be  higher  than  the  cost  of  a  non-ADS  test.  Concurrent  with  ADS  implementation, 
complexity  is  added  on  many  levels  to  test  planning,  test  rehearsal,  execution,  and  analysis.  The 
test  manager  must  be  prepared  to  include  additional  time,  money,  and  staffing  to  accommodate 
the  growth  in  test  complexity  if  ADS  is  used.  Failure  to  prepare  for  the  additional  requirements 
needed  to  implement  ADS  will  severely  diminish  the  added  benefits  of  ADS  in  overcoming  the 
performance,  environmental,  and  safety  considerations  inherent  in  traditional  T&E. 
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2. 1.2.5. 5  Ability  of  ADS  to  Overcome  the  Inadequate  Number  of  Systems  for  Interoperability 
Testing  and  Compatibility  Evaluation 

This  section  addresses  the  following  IADS  measures: 

JADS  Measure  1-2-3-7  Degree  to  which  ADS  overcomes  the  lack  of  systems  for 
interoperahility  testing  associated  with  traditional  T&E 

JADS  Measure  1-2-3-12  Degree  to  which  ADS  can  increase  the  number  of  systems  for 
compatibility  evaluation 

Traditional  tests  are  often  unable  to  perform  interoperability  testing  or  system  compatibility 
evaluations  because  of  the  limited  number  of  systems  available  on  site.  ADS  can  give  testers  the 
capability  to  overcome  these  shortfalls.  By  linking  multiple  systems  from  distributed  sites,  ADS 
tests  provide  the  opportunity  to  conduct  interoperability  testing  and  compatibility  evaluations 
that  would  be  difficult  if  not  impossible  for  most  traditional  tests. 

For  the  ETE  Test,  there  were  many  interoperability  and  compatibility  problems  among  the 
fielded  real  systems  used.  Up  to  the  time  that  the  ETE  Test  occurred,  the  target  analysis  cell 
(TAG)  had  never  had  the  opportunity  to  accomplish  interoperability  testing  or  compatibility 
evaluations  under  tactical  conditions  in  a  doctrinally  correct  manner.  During  the  ETE  Test, 
several  systems  were  able  to  operate  together  and  function  as  intended  during  combat-like 
conditions.  Specifically,  the  light  ground  station  model  (LGSM)  was  electronically  linked  to  the 
All  Source  Analysis  System  (ASAS)  workstations  located  within  the  TAG  and  provided  a 
complete  operational  picture  of  an  enemy  corps  rear  area.  In  addition,  the  ASAS  workstations 
were  electronically  linked  to  the  Advanced  Field  Artillery  Tactical  Data  System  (AFATDS) 
terminal  and  electronically  passed  fire  missions  for  prosecution  by  the  ATAGMS.  Observer 
SMEs  from  HI  Gorps  intelligence  community  stated  that  this  was  the  first  time  they  had  seen  all 
these  systems  linked  and  operating  as  they  should  during  a  conflict. 

The  EW  Test  did  not  explicitly  test  interoperability  issues  because  the  scenario  used  was  one  SPJ 
against  several  threats.  In  the  execution  of  the  EW  Test,  an  integrated  air  defense  system  (IADS) 
was  simulated  with  the  terminal  threat  hand-off  (TTH)  federate.  This  federate  designated  when 
each  threat  system  would  be  activated  against  the  target.  This  setup  did  not  specifically  test 
interoperability,  but  future  tests  could  be  changed  to  employ  an  actual  IADS  in  the  scenario. 
This  would  allow  interoperability  between  air  and  ground  forces  in  a  combat  simulation.  It  is 
also  possible  to  add  different  aircraft  to  future  EW  tests  such  as  the  Joint  Strike  Fighter  (JSF)  or 
the  F-22  to  test  interoperability  among  joint  forces  when  multiple  aircraft  are  engaging  multiple 
threats. 

The  Srr  identified  potential  uses  of  ADS  to  overcome  T&E  interoperability  shortfalls  in 
conventional  air-to-air  missile  testing.  Gonventional  missile  testing  has  a  limited  number  of  live 
shots  and  systems  because  of  cost  considerations.  This  limits  the  robust  testing  of  missiles  and 
their  associated  support  systems  in  a  variety  of  engagement  scenarios.  Architecture  similar  to 
the  srr  LFP  architecture  could  be  used  to  identify  potential  problems  between  the  shooter’s 
avionics  and  missile.  In  traditional  T&E  such  a  deficiency  would  not  be  identified  until  live 
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missiles  were  actually  fired  and  lost.  The  solution  to  such  a  problem  is  easily  implemented  and 
valuable  live  testing  could  be  saved  if  the  problem  was  identified  before  live  missiles  were  fired. 
An  ADS  architecture  like  the  SIT  LFP  architecture  could  be  used  to  identify  interoperability 
problems  by  linking  the  aircraft,  aircraft  support  systems,  and  missile  in  a  more  robust  manner 
than  a  stand-alone  lab  or  captive  carry  testing. 

2.1. 2.5.6  Ability  of  ADS  to  Facilitate  the  Use  of  Correct  Techniques  and  Procedures 
This  section  addresses  the  following  JADS  measures: 

JADS  Measure  1-2-3-24  Degree  to  which  ADS  can  insure  that  correct  techniques  and 
procedures  are  used 

JADS  Measure  1-2-3-33  Degree  to  which  ADS  can  provide  for  target  control 

JADS  Measure  1-2-3-39  Degree  to  which  ADS  can  facilitate  the  proper  use  of  friend,  foe, 
and  neutral  tactics 

In  certain  tests,  incorrect  techniques  and  procedures  may  be  used,  often  as  a  result  of  missing 
systems  or  components  and/or  restrictions  placed  upon  the  use  of  systems,  components, 
techniques,  and  procedures.  ADS  allows  for  systems  or  components  not  physically  located 
together  to  be  linked  during  testing.  This  can  enable  a  test  to  apply  the  correct  techniques  and 
procedures. 

For  example,  prior  to  the  use  of  ADS  in  the  ETE  Test,  intelligence  data  from  Joint  STARS  were 
usually  supplied  to  the  ASAS  in  the  form  of  verbal  or  written  reports.  This  was  because  without 
ADS  there  was  no  way  to  link  Joint  STARS  to  the  test  scenario  and  provide  operationally 
significant  radar  reports.  This  procedure  was  doctrinally  incorrect.  The  ground-based  portion  of 
Joint  STARS,  the  LGSM,  would  actually  have  been  electronically  linked  to  the  ASAS 
workstation  to  pass  radar  reports  to  the  analysts  working  in  the  analysis  and  control  element 
(ACE).  The  use  of  ADS  in  the  ETE  Test  enabled  the  use  of  correct  techniques  and  procedures 
for  this  combat  scenario. 

The  SIT  identified  potential  uses  for  ADS  in  improving  the  ability  to  use  correct  procedures 
during  testing  efforts.  During  conventional  live  shot  testing  of  air-to-air  missiles,  the  targets  are 
unmanned  drones.  These  drones  are  remotely  operated  and,  in  most  cases,  they  are  older  aircraft 
with  performance  capabilities  that  do  not  represent  current  threats.  The  SIT  LFP  allowed  for  the 
use  of  current  manned  aircraft  to  perform  the  engagement  scenarios.  The  use  of  manned  aircraft 
enhanced  the  ability  to  use  correct  engagement  maneuvers  for  both  target  and  shooter.  In 
addition,  ADS  may  allow  the  use  of  countermeasures  for  the  engagements.  This  use  of  correct 
techniques  with  ADS  can  improve  the  quality  of  testing  available  prior  to  actual  live  shots. 

Unfortunately,  the  EW  Test  did  not  overcome  any  use  of  incorrect  techniques  seen  in  non-ADS 
tests.  In  fact,  the  operational  staffing  of  one  threat  system  was  different  between  the  OAR  and 
AFEWES  testing  sites.  This  was  due  to  conflicting  intelligence  data  and  the  following 
implementation  of  operating  the  threat  system.  For  the  SUT,  the  most  effective  or  battle  worthy 
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ECM  technique  was  not  used  against  one  threat  to  allow  the  threat  system  an  opportunity  to 
engage  the  target.  In  a  combat  situation,  the  most  effective  ECM  technique  would  be  used  in  the 
pod  to  ensure  the  highest  effectiveness.  Furthermore,  the  rules  of  engagement  (ROE)  used  at 
each  threat  site  were  not  doctrinally  correct.  The  ROE  were  intended  to  reduce  the  variability 
effects  of  human  operators  on  the  MOP  results,  but  this  proved  ineffective.  For  future  EW  tests, 
the  difficulty  will  lie  in  deciding  which  system’s  performance  will  be  assessed.  To  assess  SUT 
performance,  it  may  be  necessary  to  regulate  threat  operator  actions  to  improve  consistency 
across  runs.  For  the  EW  Test,  it  was  very  difficult  to  simultaneously  address  both  developmental 
and  operational  test  measures  within  the  same  test  design.  However,  the  principal  interest  of 
JADS  was  assessing  the  ability  of  the  architecture  to  support  EW  testing  as  opposed  to  rigorous 
testing  of  the  SUT. 

ADS  possesses  the  potential  to  link  systems  to  operate  in  a  doctrinally  correct  manner,  but  this 
added  capability  cannot  be  realized  unless  the  test  design  fully  incorporates  and  assesses  the  use 
of  doctrinally  correct  techniques  and  procedures. 

2.1. 2.5.7  Ability  of  ADS  to  Represent  Indirect  Fire  Tactics  and  Capabilities 
This  section  addresses  the  following  JADS  measure: 

JADS  Measure  1-2-3-30  Degree  to  which  ADS  can  represent  indirect  fire  tactics  and 
capabilities 

ADS-enhanced  tests  can  realistically  portray  indirect  fire  tactics  and  capabilities.  This  can  be 
seen  by  examining  the  results  of  the  ETE  Test.  The  Phase  4  configuration  for  the  ETE  Test  used 
the  AFATDS  command  and  control  system,  collocated  with  the  TAC  at  Fort  Hood,  Texas,  to 
request  fire  missions.  TAFSM,  located  at  Fort  Sill,  Oklahoma,  processed  the  fire  missions  and 
executed  them.  When  the  ATACMS  missile  was  fired  in  TAFSM,  a  fire  protocol  data  unit 
(PDU)  was  broadcast  from  TAFSM.  The  Janus  operator,  located  at  the  Army  Training  and 
Doctrine  Command  Analysis  Center  (TRAC),  White  Sands  Missile  Range  (WSMR),  New 
Mexico,  was  alerted  that  a  mission  was  underway  so  that,  if  desired,  the  operator  could  examine 
the  strike  area  and  observe  the  effects.  At  the  appropriate  time,  a  detonate  PDU  was  issued  by 
TAFSM  giving  the  calculated  spill  location  for  the  missile’s  bomblets.  Janus  calculated  the  spill 
pattern  for  the  bomblets  and  assessed  the  appropriate  damage  to  the  targets  located  within  the 
spill’s  footprint.  Once  the  damage  was  assessed,  Janus  broadcasted  entity  state  protocol  data 
units  (ESPDU)  reflecting  the  effects  of  the  missile  strike.  This  was  a  doctrinally  correct 
representation  of  an  ATACMS  missile  strike  using  approved  missile  flyout  algorithms  and 
approved  weapons  effects  algorithms. 

The  EW  Test  system  assessed  only  a  minor  instance  of  indirect  fire  tactics  during  the  distributed 
test.  In  some  of  the  excursion  runs  executed  in  Phases  2  and  3,  the  AFEWES  site  controller 
would  call  out  simultaneous  missile  shots  from  each  weapon  system.  The  TTH  federate  also 
possessed  the  ability  to  encode  messages  for  when  specific  threat  sites  should  fire  missiles  at  the 
target.  This  capability  could  be  expanded  to  utilize  an  actual  IADS  to  add  more  realism  to  the 
test  scenario. 
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2. 1.2.5. 8  Ability  of  ADS  to  Overcome  Shortfalls  Relating  to  Threat/Friendly  Systems 
This  section  addresses  the  following  JADS  measures: 

JADS  Measure  1 -2-3-6  Degree  to  which  ADS  overcomes  traditional  T&E  shortfall  of 
inadequate  quantity  and  types  of  threat  systems 

JADS  Measure  1-2-3-8  Degree  to  which  ADS  overcomes  the  use  of  improper  threats  used  in 
traditional  T&E 

JADS  Measure  1-2-3-11  Degree  to  which  ADS  overcomes  the  traditional  T&E  shortfall  of 
inadequate  quantity  and  types  of  targets/drones 

JADS  Measure  1-2-3-14  Degree  to  which  ADS  can  provide  adequate  quantities  and  types  of 
friendly  systems 

JADS  Measure  1-2-3-21  Degree  to  which  ADS  can  provide  multidimensional  threats 

JADS  Measure  1-2-3-39  Degree  to  which  ADS  can  facilitate  the  proper  use  of  friend,  foe, 
and  neutral  tactics 

The  three  JADS  tests  provided  different  results  in  the  effort  to  overcome  the  T&E  shortfalls  of 
traditional  testing  in  the  area  of  adequate  quantity  and  quality  of  targets,  threats,  and  friendly 
systems.  Overall,  JADS  did  demonstrate  the  capability  of  ADS  to  overcome  some  of  these 
shortfalls.  However,  the  ability  of  ADS  to  overcome  shortfalls  in  this  area  is  not  inherent  to 
ADS  implementation  but  is  dependent  on  the  test  architecture  and  simulations  used. 

For  the  EW  Test,  the  addition  of  ACETEF  simulated  threat  systems  to  the  HITL  threat  systems 
used  at  AFEWES  provided  for  more  quantity  and  flexibility  than  experienced  with  threat 
systems  in  traditional  OAR  or  HTTL  testing.  Specifically,  during  Phase  3  of  the  EW  Test, 
additional  threats  were  added  in  some  runs  to  test  the  SUT  response  to  multiple  copies  of  the 
same  threat  radiating  from  different  locations.  This  allowed  the  EW  Test  team  to  assess  erratic 
behavior  of  the  SUT  when  radiated  by  multiple  signals  of  the  same  threat  type.  This  was  not 
possible  during  the  OAR  test  because  the  desired  additional  signals  were  not  available  at  the 
facility.  This  could  have  been  done  during  the  HITL  phase  if  the  JammEr  Technique  Simulator 
(JETS)  system  had  been  used  in  this  fashion.  During  Phase  3,  no  erratic  behavior  was  noted  as  a 
result  of  the  additional  threats,  suggesting  that  many  more  threats  could  be  added  to  the  scenario. 
The  addition  of  two  additional  threats  imposed  only  a  small  impact  to  the  test  setup  time.  It 
merely  required  the  implementation  and  usage  of  existing  resources  at  ACETEF.  No  testing  was 
performed  in  the  EW  Test  to  determine  the  ability  to  add  friendly  entities  to  the  test  scenario. 

The  ETE  Test  did  allow  for  the  assessment  of  additional  threat  and  friendly  systems  in  an  ADS- 
enhanced  testing  environment.  The  Janus  system  provided  the  capability  to  have  many  more 
friendly  and  threat  entities  than  ever  seen  in  a  Joint  STARS  test.  In  traditional  testing,  corps- 
level  testing  is  not  possible,  but  with  the  implementation  of  ADS-enhanced  testing,  the  total 
entity  population  was  expanded  to  nearly  10,000  entities.  The  time  and  resources  required  to 
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accomplish  this  expansion  were  considerable.  Any  similar  ADS  effort  should  be  considered 
carefully  when  deviating  from  a  traditional  test  setup.  Furthermore,  in  the  Phase  4  ETE  test, 
combining  real  assets  with  virtual  assets  used  in  the  test  scenario  also  required  considerable 
planning  and  cost.  Phase  4  relied  extensively  on  the  use  of  virtual  assets,  primarily  because  of 
the  cost  and  the  limited  availability  or  limitations .  of  live  test  assets.  These  virtual  assets 
interacted  with  other  virtual,  constructive  and  live  assets.  Future  ADS-enhanced  tests  must 
remain  flexible  when  trying  to  execute  such  tests  with  a  live/virtual  mix  of  threat  and  friendly 
systems. 

2.1. 2.5.9  Ability  of  ADS  to  Test  Integrated  Avionics 
This  section  addresses  the  following  JADS  measure; 

JADS  Measure  1-2-3-40  Degree  to  which  ADS  can  test  integrated  avionics 

In  traditional  tests,  the  ability  to  test  integrated  avionics  is  often  limited.  Conventional  test 
methods  and  emulation  of  threat  systems  with  poor  fidelity  may  be  inadequate,  resulting  in 
inconclusive  or  inaccurate  test  results.  ADS  technology  can  overcome  this  T&E  shortfall,  as 
shown  in  the  SIT,  by  providing  an  improved  capability  to  test  integrated  avionics. 

During  the  SIT,  the  launch  aircraft  avionics  were  linked  to  actual  missile  hardware  using  tactical 
hardware,  software,  and  message  protocols.  The  SIT  LSP  and  LFP  ADS  configurations  were 
found  to  be  useful  for  discrepancy/deficiency  resolution,  especially  when  there  were  interface 
issues/problems  between/among  weapon  systems  (e.g.,  the  aircraft  radar,  mission  computer, 
stores  management  system,  and  the  missile).  This  included  troubleshooting  problems  that 
proved  to  be  difficult  to  replicate  particularly  those  that  appeared  in  flight  tests  but  were  not 
readily  duplicated  in  stand-alone  laboratory  testing. 

The  Srr  ADS-enhanced  test  configurations  permitted  the  HWIL  missile  to  respond  to  actual  pre- 
and  post-launch  weapon  system  inputs  instead  of  relying  on  stand-alone  "canned"  inputs.  This 
allowed  the  HWIL  to  be  tested  in  a  more  operationally  realistic  environment  than  would  have 
been  possible  for  a  traditional  test. 
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2.1.2.5.10  Ability  of  ADS  to  Overcome  Traditional  T&E  Shortfalls  Associated  with  Modeling 
and  Simulation  (M&S)  and  Can  Provide  High-Fidelity  Emulators  and  Simulators 

This  section  addresses  the  following  JADS  measures: 

JADS  Measure  1-2-3-18  Degree  to  which  ADS  can  provide  for  improved  mod/sim  used  for 
testing 

JADS  Measure  1-2-3-20  Degree  to  which  ADS  can  overcome  mod/sim’s  inability  to 
replicate  events/entities 

JADS  Measure  1-2-3-23  Degree  to  which  ADS  can  provide  high  fidelity  emulators  and 
simulators  used  as  targets  and/or  threats 

JADS  Measure  1-2-3-26  Degree  to  which  ADS  can  facilitate  mod/sim  data  collection 

JADS  Measure  1-2-3-37  Degree  to  which  ADS  can  overcome  mod/sim  inflexibility 

ADS  technology  can  provide  vastly  improved  models  and  simulations  for  use  in  testing.  An 
inability  to  represent  desired  systems  in  a  test  environment  is  a  common  problem  in  traditional 
testing.  With  ADS,  the  tester  no  longer  has  to  have  all  M&S  assets  physically  located  in  a  single 
location  to  include  these  assets  in  testing.  This  saves  the  expense  of  purchasing  models  and 
simulations  and  the  difficulty  involved  in  manning  the  simulated  systems. 

For  the  ETE  Test,  if  the  Army  desired  to  test  any  Army  components  used  in  the  test,  they  could 
obtain  radar  reports  from  VSTARS  without  owning  and  manning  (with  Air  Force  personnel) 
their  own  VSTARS.  In  a  test  such  as  the  ETE  Test,  ADS-enhanced  testing  technology  can  ease 
the  work  required  by  linking  simulations  located  across  the  country. 

ADS  also  allows  testers  to  assemble,  using  models,  simulations,  emulators,  and  fielded  systems, 
systems  of  systems  that  either  have  not  yet  been  built  or  would  not  exist  except  in  time  of  war. 
The  ETE  Test,  for  the  first  time,  allowed  an  ATACMS  missile  to  be  fired  at  an  enemy  force 
based  on  operationally  realistic  intelligence  collected  by  Joint  STARS  and  processed  by  an 
element  of  ASAS.  This  is  just  one  example  of  a  fielded  system  of  systems.  This  ETE  Test 
scenario  also  included  the  battle  damage  assessment  provided  by  Joint  STARS  observing  the 
hulks  left  behind  and  the  fleeing  survivors.  All  these  components  were  electronically  linked  and 
functioning  as  they  would  in  actual  combat.  Using  this  distributed  testing  environment,  it  would 
be  a  simple  task  to  add  the  next  generation  ATACMS  or  ASAS  and  find  out  how  it  would 
function  as  a  component  in  this  system  of  systems. 

In  addition,  ADS  technology  is  flexible  enough  to  support  a  wide  range  of  simulation  fidelity 
requirements  depending  on  the  type  of  testing  being  accomplished.  For  example,  if  a  system 
under  test  is  conducting  early  developmental  testing,  the  fidelity  requirements  may  be 
significantly  lower  than  a  system  approaching  OT.  The  EW  Test  showed  the  possibilities  of 
linking  low-fidelity  SUTs  to  high-fidelity  threat  simulators  to  track  system  performance  through 
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the  acquisition  process.  The  EW  Test  environment  also  possessed  the  ability  to  test  a  high- 
fidelity  SUT  against  low-fidelity  threats,  as  seen  in  Phase  1  of  the  SIT. 

Lastly,  ADS  can  easily  employ  models  and  simulations  that  represent  the  threats  of  today  and  the 
threats  of  tomorrow.  All  models  and  simulations  of  threats,  no  matter  what  their  fidelity,  are 
data  driven.  The  fidelity  is  determined  by  the  accuracy  and  detail  contained  within  the  data  and 
the  fidelity  of  the  algorithms  that  use  the  data  to  depict  the  threat.  During  the  ETE  Test,  the 
threats  used  were  those  currently  fielded  within  the  Iraqi  Army.  They  could  just  as  easily  have 
been  low-observable  threats  from  a  future  battlefield.  Just  change  the  data  (radar  cross  section) 
and  the  radar  reports  change  drastically  to  reflect  a  potential  future  battlefield.  For  the  EW  Test, 
simulation  of  real-world  threat  systems  was  a  more  difficult  task.  Threat  system  data  used  in  the 
AFEWES  simulation  were  based  on  collected  intelligence  data.  As  the  collected  data  become 
more  readily  available,  the  equipment  needed  to  create  effective  and  realistic  simulators  is  the 
next  issue.  Foreign  material  is  scarcely  available  to  create  the  highest  fidelity  of  simulators. 
Furthermore,  the  operations  tactics  of  foreign  threat  systems  often  change  making  it  even  more 
difficult  to  maintain  currency  for  those  threat  systems. 

Overall,  ADS  inherently  adds  the  ability  to  link  assets  of  varied  fidelity,  but  the  creation  and 
maintenance  of  those  high-fidelity  models  are  not  necessarily  made  easier  by  the  implementation 
of  a  distributed  testing  environment. 

2. 1 .2.5. 1 1  Ability  of  ADS  to  Represent  Joint/Combined  Operations  and  Capabilities 
This  section  addresses  the  following  JADS  measure: 

JADS  Measure  1-2-3-31  Degree  to  which  ADS  can  represent  joint/combined  operations  and 
capabilities 

ADS-enhanced  tests  can  realistically  portray  joint/combined  operations.  The  Phase  4  ETE  OT 
was  a  joint  operation  that  employed  a  mix  of  Air  Force  and  Army  personnel  located  at  different 
facilities  successfully  simulating  a  C4ISR  system  interacting  with  ground-based  units.  Given 
more  of  the  necessary  planning  and  resources  needed,  ADS  could  also  represent  combined 
operations  between  forces  of  two  or  more  allies.  The  Janus  simulation  used  is  capable  of 
portraying  up  to  six  different  categories  of  forces.  Since  the  scenario  was  set  in  Iraq,  it  would 
have  been  an  easy  task  to  include  forces  from  Kuwait  or  other  North  Atlantic  Treaty 
Organization  (NATO)  forces  in  the  scenario,  thereby  creating  a  combined  operation. 

Neither  the  SIT  nor  the  EW  Test  used  joint  or  combined  operations  in  their  test  scenarios.  At 
best,  both  tests  used  multiservice  test  facilities  in  the  execution  of  the  test  phases,  but  this  did  not 
represent  combined  operations  used  in  some  traditional  tests. 
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2.1.2.5.12  Ability  of  ADS  to  Enhance  the  Analysis  Process 
This  section  addresses  the  following  JADS  measures: 

JADS  Measure  1-2-3-34  Degree  to  which  ADS  can  provide  for  real-time  analysis 

JADS  Measure  1-2-3-35  Degree  to  which  ADS  can  overcome  the  incompatibility  of 
collected  data 

This  section  examines  the  ability  of  ADS  test  participants  to  conduct  real-time  analysis  during 
testing.  Real-time  analysis  was  conducted  during  all  of  the  JADS  tests.  For  the  SIT,  using  a  live 
shooter-target  ADS  architecture  provided  the  JADS  analysts  with  immediate  feedback  on  each 
pass  of  a  multiple  pass  mission.  This  allowed  adjustments  to  be  made  to  the  remaining  test 
matrix,  if  necessary,  while  the  live  shooter  and  target  platforms  were  still  on  range.  This 
“analyst-in-the-loop”  feature  of  ADS  testing  would  be  especially  useful  in  efficiently  progressing 
through  an  ECM  testing  matrix  which  involves  varying  a  number  of  ECM-related  parameters. 
For  the  ETE  Test,  network  monitoring  tools  and  the  Janus/logger  data  collection  capabilities 
provided  data  to  JADS  analysts  allowing  for  real-time  analysis.  These  tools  provided  data  on  the 
network  links,  PDU  rates,  types  of  PDUs  being  passed,  and  the  total  number  of  PDUs  logged  at 
each  node.  Analysis  of  these  data  during  test  trials  was  important  in  ensuring  that  the  test  was 
functioning  properly  with  all  nodes  passing  the  expected  amount  and  type  of  data.  Real-time 
analysis  for  the  ETE  Test  was  limited,  however,  by  the  inability  to  manipulate  log  files  during 
testing  without  losing  test  data.  The  initial  analysis  of  log  files  had  to  be  delayed  until 
immediately  after  each  test  trial. 

During  the  EW  Test,  written  data  were  collected  on  each  active  threat  and  analyzed  immediately 
by  operators  in  the  TCAC.  This  provided  feedback  on  how  the  federation  systems  were 
performing  and  the  quality  of  the  SUT  data  being  generated.  This  real-time  analysis  required 
more  personnel  than  would  have  been  necessary  during  a  non-ADS  test  but  it  also  resulted  in 
minimizing  delays  in  acquiring  in-depth  knowledge  about  test  performance. 

The  use  of  ADS  requires  an  integrated,  systems  engineering  approach  to  the  test  planning 
process  to  insure  compatibility  of  data.  During  all  three  of  the  JADS  tests  data  had  to  be 
transformed  so  that  they  could  be  manipulated  and  analyzed  to  address  JADS  measures.  The 
JADS  test  teams  documented  the  type  and  amount  of  data  that  had  to  be  transformed  and  the 
methodology  and  tools  used  to  transfer  them.  Also  see  section  2.2.1 -2.2.2  for  further  discussion 
of  JADS  analysis  results. 
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2.1.2.5.13  Ability  of  ADS  to  Enhance  Instrumentation 
This  section  addresses  the  following  JADS  measures: 

JADS  Measure  1-2-3-5  Degree  to  which  ADS  overcomes  traditional  T&E  shortfall  of 
inadequate  instrumentation 

JADS  Measure  1-2-3-32  Degree  to  which  ADS  caaprovide  for  real-time  instrumentation 

In  general,  ADS  tests  do  little  to  overcome  instrumentation  shortfalls  of  traditional  tests.  If 
instrumentation  is  inadequate  or  incapable  of  providing  real-time  data  for  a  non-ADS  test, 
development  of  an  ADS  test  will  not  necessarily  overcome  these  problems  without  additional 
effort.  This  is  especially  true  for  ADS  tests  that  have  specific  data  transfer  requirements,  which 
may  not  be  fulfilled  by  existing  instrumentation.  Additionally,  ADS  tests  applying  a  mix  of  live 
and  virtual  assets  will  normally  require  the  same  instrumentation  for  the  live  assets  as  would  be 
required  in  traditional  testing.  ADS  testing  also  generates  instrumentation  requirements  that  are 
specific  to  distributed  architectures.  The  network  itself  must  be  extensively  instrumented  so  that 
anomalies  introduced  by  the  network  are  distinguishable  from  anomalies  associated  with  the 
SUT. 

Of  the  three  JADS  tests,  only  the  instrumentation  for  the  EW  Test  showed  a  significant 
improvement  over  similar  non-ADS  tests.  The  most  notable  improvement  of  ADS  over  non- 
ADS  instrumentation  was  in  the  data  analysis  and  visualization  instrumentation.  For  the  EW 
Test  ADS  phases,  the  instrumentation  allowed  for  much  better  insight  into  the  MOP  data  points 
as  each  run  was  executed.  The  added  insight  from  the  resulting  real-time  analysis  was  beneficial 
in  evaluating  test  performance.  Also  see  section  2.2. 1.1  for  further  discussion. 
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2.2  Concerns,  Constraints,  and  Methodologies 


The  second  JADS  issue  addresses  the  limitations  in  the  application  of  ADS  to  T&E.  JADS  used 
three  objectives  to  address  Issue  2.  Objective  2-1  addressed  characteristics  such  as 
instrumentation,  interface  performance,  bandwidth,  latency  and  data  transfer  performance  to 
determine  if  the  current  state  of  technology  provides  the  necessary  fidelity  and  is  of  the 
necessary  maturity  for  T&E  applications.  Objective  2-2  relates  to  ADS  support  systems  for  T&E 
such  as  data  management  and  analysis  systems  andxonfiguration  management.  Objective  2-3 
required  that  JADS  develop  new  and  assess  current  methodologies  for  the  use  of  ADS  in  T&E. 


Issue 

Objective 

Issue  2:  What  are  the  critical  constraints, 
concerns  and  methodologies  when  using 
ADS  for  T&E? 

Objective  2-1:  Assess  the  critical 
constraints  and  concerns  in  ADS 
performance  for  T&E. 

Objective  2-2:  Assess  the  critical 
constraints  and  concerns  in  ADS 
support  systems  for  T&E. 

Objective  2-3:  Develop  and  assess 
methodologies  associated  with  ADS 
for  T&E. 

Section  2.2. 1  below  addresses  the  critical  concerns  and  constraints  relating  to  ADS  performance 
for  T&E  applications.  To  assess  ADS  performance  JADS  looked  at  player  instrumentation  and 
interface  performance,  network  and  communications  performance  and  the  impact  of  reliability, 
availability  and  maintainability  on  T&E.  Section  2.2.2  examines  the  critical  constraints  and 
concerns  in  ADS  support  systems  for  T&E.  JADS  assessed  ADS  support  systems  for  T&E  from 
the  perspective  of  data  management  and  analysis  systems  and  configuration  management.  JADS 
Section  2.2.3  outlines  useful  procedures  for  implementing  ADS  for  T&E. 

2.2.1  Critical  Constraints,  Concerns,  in  ADS  Performance  for  T&E 

2.2. 1.1  Player  Instrumentation  and  Interface  Performance 

The  three  JADS  tests  utilized  very  different  types  of  instrumentation  in  the  execution  of  the 
various  test  phases.  For  the  EW  Test,  instrumentation  in  the  ADS  portion  was  significantly 
improved  over  the  non-ADS  tests.  This  was  mainly  in  the  form  of  instrumentation  for  data 
analysis  and  visualization.  The  non-ADS  tests  all  used  threat  site  observers  and  a  control  room 
to  view  the  overall  scenario,  but  viewing  MOP  results  was  not  possible  until  the  ADS  test 
phases.  In  these  phases,  the  Automated  Data  Reduction  Software  (ADRS)  and  the  analysis 
federate  applications  offered  much  better  insight  into  the  MOP  data  points  as  each  run  was 
executed.  These  were  very  valuable  assets  for  test  control.  Similarly,  during  the  OAR  test 
phases,  instrumentation  was  not  available  to  collect  the  jamming-to-signal  ratio  (J/S)  data  desired 
from  these  test  phases.  The  test  design  planned  to  modify  the  Radar  Detection  and  Performance 
Analysis  System  (RDAPAS)  to  collect  these  data,  but  in  the  end,  the  system  was  not  capable  of 
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collecting  the  needed  J/S  data  for  any  threat  system.  The  existing  instrumentation  used  in  the 
non-ADS  tests  was  quite  informative  and  useful,  especially  during  post-test  analysis,  but  the 
added  insight  from  the  real-time  analysis  tools  was  more  beneficial. 

The  ETE  Test  had  several  instrumentation  issues,  the  biggest  of  which  was  the  instrumentation 
of  live  assets  for  Phase  4.  This  instrumentation  was  a  major  undertaking  for  the  ETE  Test 
because  of  the  scheduling  difficulties  and  the  cost  involved.  Contrarily,  the  instrumentation  of 
the  virtual  entities  for  each  test  phase  was  exceedingly  easy  because  the  Janus  system  was 
designed  to  use  DIS  PDU  formats.  The  combination  of  the  live  instrumented  entities  and  virtual 
entities  in  the  test  environment  was  also  a  formidable  obstacle  in  test  execution,  as  the  necessary 
information  had  to  be  stripped  from  PDUs  before  being  sent  to  the  VSTARS  system  on  board  the 
E-8C  aircraft  to  minimize  the  amount  of  satellite  communications  (SATCOM)  bandwidth  used. 

The  Srr  also  experienced  difficulties  in  the  instrumentation  of  live  assets  to  be  used  in  a 
simulation  environment.  The  latency  encountered  from  the  global  positioning  system  (GPS) 
pods  on  the  shooter  and  target  aircraft  severely  limited  the  intended  scenarios  available  for  use. 
Because  timely  instrumentation  of  the  aircraft  positions  linked  to  the  missile  simulator  w^ 
troublesome,  very  few  maneuvering  flight  scenarios  could  be  used  during  test  execution.  In  this 
case,  the  problem  for  SIT  was  not  the  ability  to  instrument  but  rather  the  ability  to  instrument  and 
pass  the  required  data  to  the  appropriate  destinations  in  a  timely  fashion. 

Overall,  ADS  does  little  to  overcome  instrumentation  shortfalls  of  the  T&E  community.  If 
instrumentation  is  not  available  for  a  non-  ADS  test,  developing  an  ADS-enhanced  test  will 
usually  not  overcome  this  problem.  In  addition,  in  cases  such  as  SIT,  available  instrumentation 
may  not  be  able  to  pass  the  required  data  to  the  destination  in  a  timely  fashion,  further  limiting 
the  usefulness  of  an  ADS-enhanced  test.  Instrumentation  problems  are  not  usually  solved  by 
placing  the  components  in  an  ADS  environment,  but  doing  this  may  exacerbate  any  problems 
currently  existing  in  this  area. 

2. 2. 1.2  Network  and  Communications  Performance 

One  of  the  primary  purposes  of  IADS  testing  was  to  determine  if  current  network  and  high-speed 
data  communications  structures  were  capable  of  supporting  testing  and  training  in  an  ADS 
environment.  Specifically,  JADS  assessed  the  network  capability  in  three  key  areas:  adequate 
bandwidth  to  transfer  large  quantities  of  data  from  one  node  to  another;  the  ability  to  transfer 
these  data  from  one  entity  or  host  to  another  with  minimal  delays  (latency);  and  the  ability  to 
transfer  the  data  completely  and  accurately  (minimal  data  losses/corrupt  data).  The  means  used 
to  test  these  network  and  communications  systems  capabilities  and  the  performance  results  are 
discussed  below. 

Bandwidth 

The  three  JADS  tests  used  multiple  methods  to  transfer  data  from  one  node  to  another.  These 
included  T-1,  T-3  lines,  and  RF  satellite  capabilities.  The  use  of  each  data  transfer  method  and 
its  bandwidth  characteristics  is  discussed  below. 
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The  ETE  and  EW  tests  used  T-1  lines  to  connect  different  nodes  and  transmit  data.  The  T-1 
lines  have  a  normal  bandwidth  of  1.544  megabits  per  second  (Mbps).  A  small  portion  of  this 
bandwidth  was  partitioned  off  to  provide  a  secure  voice  channel  during  both  tests  leaving  a 
bandwidth  of  1 .344  Mbps  to  transmit  data.  SPECTRUM®  monitored  bandwidth  based  on  what 
was  left  for  data.  Network  link  performance  data  from  the  T-1  line,  including  packet  rate  and 
percentage  of  bandwidth  utilized,  were  collected  and  studied  using  the  SPECTRUM  network 
analysis  tool.  A  polling  rate  of  15  seconds  was  used  for  the  ETE  Test  and  a  polling  rate  of  30 
seconds  was  used  for  the  EW  Test.  Once  all  the  data  were  gathered,  the  JADS  analysts 
consolidated  the  data  by  network  link.  These  data  were  then  used  to  calculate  daily  packet  rate 
and  bandwidth  values  (maximum  and  average)  for  each  link.  The  bandwidth  values  were 
provided  by  SPECTRUM  as  the  percentage  of  bandwidth  available  on  the  T-1  line. 

The  ETE  Test  also  used  a  SATCOM  link  to  transfer  stripped  PDU  data  from  the  ground  data 
terminal  (GOT)  to  the  air  data  terminal  (ADT)  on  the  E-8C.  Because  of  the  limited  capacity  of 
the  SATCOM  link,  the  PDUs  were  stripped  from  1156  bytes  to  192  bytes  to  minimize  data 
losses  that  could  have  resulted  from  bandwidth  restrictions.  Because  of  the  limited  measurement 
capabilities  on  the  E-8C  aircraft,  no  bandwidth  data  could  be  collected  for  this  link. 

As  was  the  case  with  the  ETE  and  EW  tests,  the  LSP  phase  of  the  SIT  employed  T-1  lines  to 
connect  nodes  and  transfer  data.  Each  simulation  node  (Weapon  System  Support  Facility 
{WSSF],  Weapons  System  Integration  Center  [WSIC],  Simulation  Laboratory  [SIMLAB])  was 
DIS  compliant  and  encrypted.  The  network  data  exchange  protocol  was  DIS  Version  2.0.4, 
except  for  the  stores  management  system  (SMS)  data  exchange  between  the  F/A-18  WSSF  and 
the  AIM-9  SIMLAB.  ESPDUs  were  generated  at  each  player’s  node  and  passed  to  all  other 
nodes  via  T-1  lines  for  use  as  required  at  those  nodes.  A  T-1  link  between  the  nodes  and  the 
JADS  TCAC  allowed  JADS  personnel  to  monitor  and  control  the  simulated  intercepts. 

In  the  LFP,  two  live  F-16  fighter  aircraft  flying  over  a  range  were  linked  to  the  AMRAAM  AIM- 
120  HWIL  simulation  facility.  GPS  and  telemetry  data  were  downlinked  from  the  aircraft  and 
combined  by  the  time-space-position  information  (TSPI)  data  processor  (TDP)  to  produce 
optimal  entity  state  solutions.  The  aircraft  entity  state  data  were  transformed  into  DIS  PDUs  and 
transferred  to  the  AMRAAM  HWIL  laboratory  over  a  T-3  link.  The  shooter  aircraft  “fired”  the 
AMRAAM  in  the  MISILAB  at  the  target  and  provided  data  link  updates  of  the  target  position 
and  velocity  to  the  missile  during  its  flyout.  The  AMRAAM  seeker  was  mounted  on  a  flight 
table  and  responded  to  RF  sources  in  the  Missile  Simulation  Laboratory  (MISILAB)  which 
simulated  the  seeker  return  from  the  target,  the  relative  motions  of  the  target  and  the  missile,  and 
ECM.  A  T-1  link  between  the  Central  Control  Facility  (CCF)  and  the  JADS  TCAC  allowed 
JADS  personnel  to  monitor  the  simulated  intercepts. 

Prior  to  the  installation  of  the  network,  the  SIT  analysts  estimated  the  expected  PDU  traffic  from 
each  site.  Along  with  this  expected  PDU  traffic,  any  additional  network  traffic  was  considered 
in  determining  the  hardware  requirements.  After  the  initial  installation,  baseline  testing  was 
conducted  by  sending  PDUs  across  the  network.  During  the  actual  test,  JADS  analysts 
monitored  the  bandwidth  usage  with  SPECTRUM  software  tools. 
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The  average  and  peak  packet  rates  and  load  values  experienced  for  each  of  the  three  JADS  tests 
are  presented  in  the  following  tables.  These  tables  refer  only  to  active  test  time  during  which 
PDU  loggers  were  recording  data. 

Table  6.  ETE  Test  Bandwidth  Usage 


Phase 

Node  A 

NodeB 

Lx)ad 

Average  Pe^ 

Packet  Rate 

Average  Peak 

2 

T 

G 

0.58% 

20.0% 

17.92/sec 

120.0/sec 

G 

H 

1.07% 

5.0% 

33.74/sec 

102.0/sec 

3 

T 

G 

.58% 

20.0% 

17.92/sec 

120.0/sec 

G 

H 

1.07% 

5.0% 

33.74/sec 

102.0/sec 

4 

T 

G 

.6% 

*69% 

15.1/sec 

345.0/sec 

G 

H 

.3% 

4% 

15.5/sec 

93.0/sec 

Total 

T 

G 

0.59% 

20.0% 

G 

H 

.81% 

5.0% 

27.66/sec 

102.0/sec 

G  =  Northrop  Gruraman  H  =  Fort  Hood  T  —  TCAC 

*The  peak  bandwidth  of  69%  was  due  to  non-test  software  development  activities  at  the  TCAC. 
These  activities  were  halted  once  the  effect  on  bandwidth  was  discovered. 


Table  7.  EW  Test  Bandwidth  Usage 


Phase 

Node  A 

Node  B 

Load 

Average  Peak 

Packet  Rate 

Average  Peak 

2 

JADS 

AFW 

6.75% 

27% 

54.33/sec 

198/sec 

JADS 

ACE 

2.96% 

11% 

28.68/sec 

103/sec 

AFW 

ACE 

4.18% 

19% 

32.95/sec 

138/sec 

3 

JADS 

AFW 

5.53% 

65% 

46.21/sec 

315/sec 

JADS 

ACE 

2.73% 

10% 

26.74/sec 

85/sec 

AFW 

ACE 

2.51% 

21% 

20.67/sec 

151/sec 

Total 

JADS 

AFW 

6.14% 

65% 

50.27/sec 

315/sec 

JADS 

ACE 

2.85% 

11% 

27.71/sec 

103/sec 

AFW 

ACE 

3.35% 

21% 

26.81/sec 

151/sec 

AFW  =  AFEWES  ACE  =  ACETEF 

*The  peak  packet  rate  and  utilized  bandwidth  values  were  all  captured  on  the  fourth  day  of  test 
execution  while  excursions  were  being  run  with  four  active  threats. 


54 


Table  8.  SIT  Bandwidth  Usage 


Phase 

Network 

Available 

Bandwidth 

(Mbps) 

Max  Load 

Max  Packet 
Rate 

LSP 

Point  Mugu- 
China  Lake  (T-1) 

2.048 

4% 

Not 

Measured 

WSSF-SIMLAB 

(LAN) 

1.536 

2% 

Not 

Measured 

LFP 

CCF-MISILAB 

44.736 

<1% 

201 /sec 

CCF-TCAC 

1.457 

<3% 

49/sec 

LAN  =  local  area  network 


Conclusion 

T-1  lines  used  by  JADS  for  all  three  tests  provided  more  than  adequate  bandwidth  to  conduct 
ADS-enhanced  tests.  None  of  the  tests  were  delayed  or  negatively  impacted  because  of 
bandwidth  constraints.  The  average  load  for  all  the  tests  was  typically  less  than  10  percent.  The 
peak  load  experienced  by  any  of  the  tests  was  69  percent  and  65  percent  of  the  available 
bandwidth  during  the  ETE  and  EW  tests,  respectively.  The  ETE  Test  peak  load  was  a  result  of 
non-test  related  activities  and  was  resolved  before  the  completion  of  Phase  4.  The  EW  Test  peak 
load  was  measured  during  Phase  3;  this  peak  occurred  while  four  threat  systems  were  active 
simultaneously  during  excursion  runs.  Neither  peak  load  had  a  negative  impact  on  the  test. 
Likewise,  the  bandwidth  constraints  of  the  SATCOM  link  for  the  ETE  Test  did  not  have  a 
negative  impact.  Stripping  the  PDUs  to  include  only  the  essential  bytes  relieved  the  potential 
problems  that  could  have  resulted  from  the  limited  SATCOM  link  bandwidth.  For  the  SIT, 
bandwidth  problems  were  not  experienced  during  either  the  LSP  or  LFP.  The  bandwidth  usage 
was  well  within  expected  levels. 

Latency 

All  the  electronic  data  (PDUs,  electronic  messages)  transmitted  via  T-1  lines  or  SATCOM  links 
were  time  stamped  and  recorded  at  the  transmitting  and  receiving  node  using  loggers  (developed 
by  JADS  analysts  and  contract  agencies).  The  loggers  specifically  recorded  the  time  and  order 
that  the  PDUs/electronic  messages  were  transmitted  and  received  at  each  node.  The  data  were 
later  retrieved  and  correlated  by  JADS  analysts  to  determine  the  latency  (transmission  time  from 
node  to  node).  These  calculations  were  accomplished  using  UNDC'^^’^-based  software  tools 
created  by  JADS  programmers.  The  results  are  as  follows. 
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Table  9.  EXE  Test  Latency  Data 


Phase 

Node  A 

Node  B 

Latency  (seconds)  I 

Minimum 

Mean 

Maximum 

2 

W 

T 

0.016 

0.039 

0.145 

T 

G 

0.050 

0.057 

1.090 

S 

W 

0.030 

0.035 

0.396 

3 

W 

T 

.020 

.047 

.172 

T 

G 

-1.07* 

-.42* 

.152 

S 

W 

.036 

.038 

.375 

4 

w 

T 

.017 

.046 

.178 

T 

G 

.039 

.058 

.291 

S 

W 

.035 

.038 

.399 

Ground  NIU 

Air  NIU 

1.58 

12.56 

85.57 

Total 

W 

T 

0.016 

0.044 

0.178 

T 

G 

0.039 

0.058 

1.090 

S 

W 

0.030 

0.037 

0.399 

Ground  NIU 

Air  NIU 

1.58 

12.56 

85.57 

G  =  Northrop  Grumman  NIU  =  networic  interface  unit  S  =  Fort  Sill 

T  =  TCAC  W  =  WSMR 


*During  Phase  3,  logger  clocks  could  not  be  synchronized  at  the  Grumman  node  because  of  a 
problem  with  the  time  synchronization  program.  This  resulted  in  negative  latencies  (not 
included  in  the  calculations  for  the  total  column).  This  time  synchronization  problem  was 
resolved  during  Phase  3. 


For  both  ADS  phases  of  the  EW  Test,  node-to-node  latency  values  across  relevant  network  links 
were  evaluated  for  six  complex  data  message  types.  The  six  types  were  selected  based  on  their 
ability  to  provide  insight  into  the  impact  of  latent  traffic  on  SUT  data  validity. 
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Table  10.  EW  Test  Phase  2  Latency  Data  (in  milliseconds) 


DATA  ELEMENT 

TYPE 

J/U)S- 

AFEWES 

JADS- 

ACETEF 

AFEWES- 

ACETEF 

LiveEntityState 
(A/C  TSPI) 

UDP 

Avg:  43.9 
Max:  859 

Avg:  41.2 
Max:  861 

N/A 

Threat  Performance 
(Threat  Track  Data) 

UDP 

Avg:  45.9- 
Max:  860 

Avg:  42.6 
Max:  861 

N/A 

Threat  Performance 
(T/E,  J/S,  Target 
Location) 

UDP 

Avg:  32.1 
Max:  7975 

N/A 

Avg:  35.7 
Max:  8540 

SUT_Jammer  _Tech 
(DSM  RF  Emissions) 

TCP 

N/A 

Avg:  130 
Max:  13951 

Avg:  104.3 
Max:  9680 

SUT_Receiver_T  rack 
(Verify  Environment) 

TCP 

N/A 

Avg:  112.7 
Max:  13982 

N/A 

Source_Mode 

Change 

(Threat  RF  Emission) 

TCP 

Avg:  101 
Max:  9556 

Avg:  66 
Max:  8022 

Avg:  61 

Max:  7701 

A/C  =  aircraft  T/E  =  tracking  error  TCP  =  transmission  control  protocol  UDP  =  user  datagram  protocol 


Table  11.  EW  Test  Phase  3  Latency  Data  (in  milliseconds) 


DATA  ELEMENT 

TYPE 

JADS- 

AFEWES 

JADS- 

ACETEF 

AFEWES- 

ACETEF 

LiveEntityState 
(A/C  TSPI) 

UDP 

Avg:  45.7 
Max:  1150 

Avg:  44.4 
Max:  472 

N/A 

Threat  Performance 
(Threat  Track  Data) 

UDP 

Avg:  52.7 
Max:  1151 

Avg:  52.3 
Max:  516 

N/A 

Threat  Performance 
(T/E,  J/S,  Target 
Location) 

UDP 

Avg:  30.0 
Max:  515 

N/A 

Avg:  41.2 
Max:  511 

SUT_Jammer  _Tech 
(DSM  RF  Emissions) 

TCP 

N/A 

Avg:  75.3 
Max:  312 

Avg:  67.3 
Max:  296 

SUT_Receiver_T  rack 
(Verify  Environment) 

TCP 

N/A 

Avg:  77.0 
Max:  267 

N/A 

Source_Mode 

Change 

(Threat  RF  Emission) 

TCP 

Avg:  55.7 
Max:  372 

Avg:  71.9 
Max:  1548 

Avg:  45.5 
Max:  501 

A/C  =  aircraft  T/E  =  tracking  error  TCP  =  transmission  control  protocol  UDP  =  user  datagram  protocol 


For  the  SIT,  the  latencies  of  all  PDU  traffic  were  monitored  and  recorded  by  JADS  loggers. 
Table  12  shows  the  no  de-to-node  latency  for  the  LSP.  Table  13  shows  the  node-to-node  latency 
for  the  LFP.  The  latencies  were  measured  across  T-1  or  dedicated  networks  for  LSP  and  T-1  or 
T-3  for  the  LFP. 


Table  12.  SIT  LSP  Latency  Data 


Phase 

Node  A 

NodeB 

Latency  ( in 
milliseconds) 

Mean 

Std  Dev 

LSP 

WSIC 

SIMLAB 

17.2 

8.3 

SMLAB 

WSIC  * 

22.3 

18.3 

WSSF 

WSIC 

19.7 

15.3 

WSIC 

WSSF 

19.2 

8.1 

Table  13.  SIT  LFP  Latency  Data 


Phase 

Node  A 

Node  B 

Latency  ( in  milliseconds) 

Mean 

Std  Dev 

LFP 

CCF 

MISILAB 

1.2 

0.9 

MISILAB 

CCF 

8.2 

4.2 

CCF 

TCAC 

28.6 

5.0 

Conclusion 

The  tables  show  that  the  test  networks  were  very  effective  in  maintaining  stable  latencies  during 
the  JADS  tests.  The  only  high  latency  issues  for  the  ETE  Test  were  associated  with  the 
SATCOM  link.  These  high  latencies  were  the  result  of  the  numerous  buffers  required  on  the 
E-8C  to  process  the  test  data  and  were  beyond  the  control  of  JADS  personnel. 

For  the  EW  Test,  seven  percent  and  four  percent  of  the  runs  for  Phases  2  and  3,  respectively, 
experienced  high  latencies.  These  anomalies  were  carefully  researched  to  determine  the  impact 
of  these  latencies.  Following  this  research,  JADS  analysts  determined  that  the  high  latencies  had 
almost  no  impact  on  the  large  majority  of  the  runs,  with  only  one  run  from  Phase  2  excluded 
from  the  valid  SUT  data  set  and  no  Phase  3  runs  excluded. 

The  latencies  observed  over  the  network  for  the  SIT  were  as  predicted  by  baseline  testing  of  the 
network  after  the  initial  installation.  The  biggest  contributor  to  latency  was  not  the  length  of  the 
network  but  rather  how  many  interfaces  the  network  traffic  was  passing  through.  The  network 
latencies  for  both  phases  of  the  SIT  were  minor  compared  to  the  latencies  observed  when  the 
individual  simulators  received  and  processed  the  PDUs  from  the  network. 

In  general,  the  impact  of  high  latencies  on  ADS-enhanced  tests  is  dependent  upon  the  type  of  test 
executed.  Tests  such  as  the  ETE  Test  can  sustain  high  latencies  (even  latencies  as  high  as  those 
experienced  over  the  SATCOM  link)  and  have  no  negative  impact  on  test  execution  because  of 
the  inherent  delays  involved  in  human-in-the-loop  systems.  However,  in  the  SIT  and  EW  Test, 
high  latencies  could  have  caused  serious  problems  and  result  in  unusable  trials.  Test  directors 
must  consider  these  latency  issues  during  test  planning  and  determine  the  level  of  latency 
acceptable  for  their  test. 
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Data  Transfer  Performance 


The  performance  of  the  network  and  communications  systems  in  transferring  data  was  measured 
and  analyzed  for  all  the  JADS  tests.  Specifically,  the  ability  of  these  systems  to  transfer  data 
completely  and  accurately  with  a  minimal  loss  or  corruption  of  data  was  the  focus.  This 
measurement  and  analysis  was  accomplished  through  the  use  of  the  log  files  created  by  the 
JADS  loggers  and  the  analysis  tools  created  by  the  JADS  program  analysts.  The  results  of  the 
data  transfer  performance  for  the  three  tests  are  displayed  in  the  following  tables. 

For  the  ETE  Test,  the  effectiveness  of  the  network  and  SATCOM  link  at  passing  data  was 
evaluated  through  manipulation  and  comparison  of  PDU  log  files  from  each  test  location.  This 
analysis  was  only  possible  because  of  the  software  tools  created  by  the  JADS  program  analysts. 
Log  files  containing  more  than  100,000  PDUs  could  be  examined  in  minutes,  a  task  that  would 
have  taken  days  without  the  software  tools. 

Table  14  shows  the  summary  PDU  data  for  each  ETE  Test  phase  by  node  including  the 
SATCOM  data  from  the  live  flight  trials.  No  PDU  loss  rate,  for  any  phase  or  the  ETE  Test  as  a 
whole,  exceeded  three  percent  of  the  PDUs  sent.  In  addition,  no  corrupt  (duplicate  or  out  of 
order)  PDUs  were  received.  This  shows  the  effectiveness  of  the  network  and  SATCOM  link  at 
passing  PDUs  between  ETE  Test  nodes. 

For  the  EW  Test,  the  effectiveness  of  data  transfer  across  network  links  was  measured  and 
analyzed  in  terms  of  six  complex  data  message  types.  The  six  types  were  selected  based  on  their 
ability  to  provide  insight  into  the  impact  of  lost  traffic  on  SUT  data  validity.  In  other  words, 
these  were  the  message  types  that  if  lost  should  have  had  the  most  noticeable  effect  on  SUT 
behavior  and  SUT  performance  measure  data. 
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Table  14.  ETE  Test  PDU  Data 


Phase 

Node  A 

Node  B 

PDUs  Sent 

PDUs 

Received 

PDUs  Lost/ 
Percent  Lost 

Corrupt 

PDUs 

W 

T 

382,254 

382,159 

95 

.025% 

0 

2 

T 

G 

382,159 

381,959 

200 

.052% 

0 

S 

W 

13,745 

13,744 

1 

.007% 

0 

w 

T 

226,440 

220,210 

6,230 

2.75% 

0 

3 

T 

G 

220,210 

219,159 

1,051 

0.477% 

0 

S 

W 

4,986 

87 

1.71% 

0 

W 

T 

953,456 

947,442 

6,014 

0.63% 

0 

4 

T 

G 

947,442 

944,002 

3,440 

0.36% 

0 

S 

W 

30,089 

30,077 

12 

0.04% 

0 

NIU 

E-8C 

359,144 

349,297 

9,847 

2.74% 

0 

W 

T 

1,562,150 

1,549,811 

0 

Total 

T 

G 

1,549,811 

1,545,120 

4,691 

0.30 

0 

S 

W 

48,907 

48,807 

100 

0.20% 

0 

NIU 

E-8C 

359,144 

349,297 

9,847 

2.74% 

0 

E-8C  =  aircraft  G  =  Northrop  Grumman  NIU  =  ground  network  interface  unit  at  Grumman 

S  =  Fort  Sill  T  =  TCAC  W=  WSMR 
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Table  15.  EW  Test  Phase  2  Lost  Data  Traffic  Messages  by  Link 


A/C 


DATA  ELEMENT 

TYPE 

JADS-AFEWES 

JADS-ACETEF 

AFEWES- 

ACETEF 

LiveEntityState 
(A/C  TSPI) 

UDP 

Avg  Lost:  6.9 

Avg  Sent:  4000 
Percent  Lost;  <  1 
Max:  503 

Avg  Lost:  18.6 

Avg  Sent:  4000 
Percent  Lost:  <  1 
Max:  1266 

N/A 

Threat  Performance 
(Threat  Track  Data) 

UDP 

Avg  Lost:  6.3 

Avg  Sent:  4000 
Percent  Lost:  <  1 
Max:  504 

Avg  Lost:  18.1 

Avg  Sent;  4000 
Percent  Lost:  <  1 
Max;  1266 

N/A 

Threat  Performance 
(T/E,  J/S,  Target 
Location) 

UDP 

Avg  Lost:  4.2 

Avg  Sent;  4000 
Percent  Lost:  <  1 
Max:  182 

N/A 

Avg  Lost:  14.8 

Avg  Sent:  4000 
Percent  Lost:  <  1 
Max:  2222 

SUT_Jammer  _Tech 
(DSM  RF  Emissions) 

TCP 

N/A 

Avg  Lost:  0 

Avg  Sent:  9 
Percent  Lost:  0 
Max:  0 

Avg  Lost:  0 

Avg  Sent:  9 

Percent  Lost:  0 

Max:  0 

SUT_Receiver_Track 
(Verify  Environment) 

TCP 

N/A 

Avg  Lost;  0 

Avg  Sent:  96 
Percent  Lost:  0 
Max:  0 

N/A 

Source_Mode 

Change 

(Threat  RF  Emission) 

TCP 

Avg  Lost:  0 

Avg  Sent:  50  -90 
Percent  Lost:  0 
Max:  0 

N/A 

Avg  Lost:  0 

Avg  Sent;  50  -  90 
Percent  Lost:  0 

Max:  0 

aircraft  T/E  =  tracking  error  TCP  =  transmission  control  protocol  UDP  =  user  datagram  protocol 
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Table  16.  EW  Test  Phase  3  Lost  Data  Traffic  Messages  by  Link 


DATA  ELEMENT 

TYP 

E 

JADS-AFEWES 

JADS-ACETEF 

AFEWES- 

ACETEF 

LiveEntityState 
(A/C  TSPI) 

UDP 

AvgLost:  .08 

Avg  Sent:  3900 
Percent  Lost:  <  1 
Max:  17 

AvgLost:  0 

Avg  Sent:  3900 
Percent  Lost:  0 
Max:  0 

N/A 

Threat  Performance 
(Threat  Track  Data) 

UDP 

AvgLost:  .08 

Avg  Sent:  3800 
Percent  Lost:  <  1 
Max:  17 

AvgLost:  0 

Avg  Sent:  3800 
Percent  Lost:  0 
Max:  0 

Threat  Performance 
(T/E,  J/S,  Target 
Location) 

UDP 

AvgLost:  .2 

Avg  Sent:  3900 
Percent  Lost:  <  1 
Max:  36 

N/A 

AvgLost:  .002 
Avg  Sent:  3900 
Percent  Lost:  <  1 
Max:  1 

SUT_Jammer  _Tech 
(DSM  RE  Emissions) 

TCP 

N/A 

AvgLost:  .006 
Avg  Sent:  10 
Percent  Lost:  <  1 
Max:  1 

AvgLost:  .006 
Avg  Sent:  10 
Percent  Lost:  <1 
Max:  1 

SUT_Receiver_Track 
(Verify  Environment) 

TCP 

N/A 

Avg  Lost:  0 

Avg  Sent:  116 
Percent  Lost:  0 
Max:  0 

N/A 

Source_Mode  Change 
(Threat  RF  Emission) 

TCP 

Avg  Lost:  0 

Avg  Sent:  29 
Percent  Lost:  0 

Max:  0 

N/A 

AvgLost:  0 

Avg  Sent:  29 
Percent  Lost:  0 
Max:  0 

A/C  =  aircraft  T/E  =  tracking  error  TCP  transmission  control  protocol  UDP  -  user  datagram  protocol 

The  network  was  very  reliable  in  transferring  data  during  the  EW  Test.  No  transmission  control 
protocol  (TCP)  (reliable)  data  traffic  losses  were  detected  during  Phase  2  and  only  one  unique 
TCP  reliable  data  traffic  loss  was  found  during  Phase  3  (JADS  analysts  attribute  responsibility 
for  the  lost  TCP  message  to  the  runtime  infrastructure  (RTI)  itself,  but  the  specific  cause  could 
not  be  pinpointed).  In  addition,  less  than  one  percent  of  all  user  datagram  protocol  (UDP)  (best 
effort)  data  traffic  was  lost  in  Phases  2  and  3.  Only  one  run  was  excluded  from  the  SUT  valid 
data  set  in  Phase  2,  with  no  Phase  3  runs  excluded. 


During  the  SIT,  PDU  loggers  were  used  to  measure  any  missing,  out-of-order,  or  corrupted 
PDUs.  There  were  no  instances  of  these  problems  for  either  the  LFP  or  the  LSP.  The  only  noted 
problems  were  repeated  shooter  or  missile  PDUs  during  the  LSP.  These  repeating  PDUs  were 
generated  by  the  network  interface  units  (NIUs)  for  unknown  reasons.  The  repeaters  caused 
some  problems  with  dead  reckoning  of  entity  locations  for  display  purposes.  However,  no  target 
PDUs  repeated,  so  no  problems  were  experienced  within  the  simulations  and  execution  of  the 
engagements. 
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Conclusion 


Overall,  the  data  transfer  performance  of  the  networks  for  the  three  JADS  tests  was  excellent. 
The  problems  experienced  were  minor  and  did  not  negatively  impact  the  success  of  the  tests. 

In  addition,  any  concerns  about  the  impact  of  the  relationships  among  data  transfer  performance, 
bandwidth,  and  latency  were  dismissed  because  of  the  results  reported  for  the  three  tests. 
Neither  the  peak  bandwidth  nor  latency  appeared  to  have  any  impact  on  the  ability  of  the 
network  to  pass  data  reliably  and  accurately  in  an  ADS-enhanced  test  environment.  However, 
for  distributed  tests  requiring  a  greater  amount  of  data  processing  in  a  shorter  test  period,  it  is 
possible  that  the  relationships  of  these  network  characteristics  could  become  a  concern. 

2.2.1.3  Impact  of  Reliability,  Availability,  and  Maintainability  on  T&E 

The  ability  of  ADS  systems  to  be  up  and  operating  at  scheduled  test  initialization  and  to  remain 
operational  throughout  the  test  period  is  fundamental  to  the  success  of  a  distributed  test. 
Distributed  testing  systems  experiencing  low  levels  of  reliability,  maintainability,  and 
availability  (RM&A)  can  cause  test  trials  to  be  delayed,  rescheduled,  repeated,  or  canceled 
completely.  For  the  three  JADS  tests,  execution  logs  were  maintained  at  each  node  to  document 
the  performance  of  the  ADS  systems  involved  and  the  overall  status  of  the  test  trials.  Following 
each  test  phase,  system  malfunctions  and  failures  were  examined  in  each  area  in  terms  of  lost  test 
time  and  loss  of  usable  test  trial  data.  This  analysis  provided  insight  which,  in  several  cases, 
enabled  system  improvements  to  be  made  prior  to  the  next  phase  of  testing. 

The  following  paragraphs  and  tables  summarize  the  impacts  of  ADS  system  RM&A  behavior 
across  the  three  JADS  tests  in  terms  of  test  time  and  trial  losses.  First,  ADS  system  hardware 
and  software  problems  are  presented  exclusive  of  the  network,  personnel,  and  procedural 
problems  that  impacted  testing.  Detailed  discussion  of  the  problems  in  those  areas  follows. 
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ADS  System  Hardware  and  Software 


Table  17.  Test  Trials  Canceled/Aborted  Because  of  ADS  System  Failures 


Test 

Phase 

Total  # 
of  Trials 

#  of  Trials 

Lost  Because  of 
ADS  System 
Failures 

%  of  Trials 

Lost  Because  of  ADS 
System  Failures 

ETE 

2 

5 

0 

0% 

3 

7 

3 

43  % 

4 

9 

1 

11  % 

EW 

2 

363  n 

83 

23  % 

3 

270 

21 

8  % 

srr 

LSP 

214 

30 

14% 

LFP 

37 

0 

0% 

As  can  be  seen  in  Table  17,  the  ETE  Test  experienced  the  cancellation  of  four  trials  because  of 
ADS  system  failures  during  Phases  3  and  4.  Of  the  four  trials  lost,  two  were  due  to  problems 
with  VSTARS  and  two  were  due  to  the  surveillance  control  data  link  (SCDL)  at  Northrop 
Grumman.  In  addition  to  these  failures,  15  minor  ADS  system  failures  were  experienced  during 
Phases  2  through  4.  These  failures  did  not  involve  critical  systems  and  did  not  result  in  a  break 
in  the  ETE  Test  loop,  thus  avoiding  any  delays  or  cancellation  of  the  trials. 

Unlike  each  ETE  Test  trial,  which  filled  an  entire  day’s  testing,  an  individual  EW  Test  or  SIT 
trial  required  only  minutes  to  complete.  Thus,  when  an  EW  Test  or  SIT  trial  had  to  be  aborted 
because  of  a  system  malfunction  or  failure,  the  particular  scenario  being  run  was  usually 
repeated  immediately.  This  strategy  for  accomplishing  runs  allowed  test  controllers  to  monitor 
the  overall  test  progress  and  the  amount  of  usable  data  that  had  been  collected. 

Because  of  the  nature  of  the  EW  Test  and  the  short  duration  of  each  individual  trial,  looking  at 
the  number  of  lost  test  trials  does  not  provide  a  complete  picture  of  test  federation  RM&A 
performance.  Each  aborted  trial  resulted  in  lost  data  but  not  in  a  significant  amount  of  lost  test 
time.  Table  17  provides  additional  insight  into  test  federation  RM&A  by  capturing  time  during 
which  no  active  test  trials  were  even  attempted.  This  is  time  during  which  equipment  was  being 
troubleshot  and  maintained  or  during  which  discussion  about  malfunctioning  hardware  and 
software  delayed  the  restart  of  active  trials.  Operators  were  typically  released  on  break  for  a 
short  time  if  the  ADS  system  problem  was  deemed  severe  enough  to  require  more  than  ten 
minutes  of  maintenance  activity.  As  the  table  shows,  both  phases  resulted  in  close  to  the  same 
number  of  hours  being  lost  over  nine  test  days.  The  main  contributors  to  lost  test  time  in  Phase  2 
were  the  hardware  and  software  systems  implemented  to  enable  distribution,  whereas  the  major 
problems  experienced  in  Phase  3  were  with  existing  test  facility  systems— problems  which  took 
some  time  to  diagnose  and  fix. 

EW  Test  Phase  2  RM&A  analysis  showed  that,  aside  from  the  great  deal  of  test  time  lost  because 
of  existing  equipment  malfunctions  at  one  of  the  test  facilities,  the  majority  of  lost  test  trials  were 


caused  by  federation  and  visualization  tool  software  crashes.  Based  on  this  information, 
numerous  problems  were  identified  and  fixed  through  software  upgrades  prior  to  Phase  3.  These 
fixes  were  responsible  for  the  significant  drop  (from  23  percent  to  only  8  percent)  in  the  number 
of  trials  lost  to  these  problems,  as  shown  in  Table  18.  Thus,  far  fewer  total  runs  were  required  in 
Phase  3  to  obtain  a  satisfactory  amount  of  usable  SUT  data. 

Table  18.  Time  Lost  Because  of  ADS  System  Failures 


Test 

Phase 

Total  Test 
Time 

Total  Time  Lost 
Because  of  ADS 
System  Failures 

%  of  Time  Lost 
Because  of  ADS 
System  Failures 

ETE 

2 

35  hrs,  27  mins 

0  mins 

0% 

3 

35  hrs,  13  mins 

21  hrs 

60% 

4 

57  hrs,  9  mins 

8  hrs 

14% 

EW 

2 

57  hrs,  12  mins 

7  hrs,  18  mins 

13  % 

3 

48  hrs,  48  mins 

10  hrs 

21  % 

SIT 

LSP 

19  hrs,  42  mins 

2  hrs,  52  mins 

15% 

LFP 

3  hrs,  39  mins 

0  mins 

0% 

The  ETE  Test  time  data  in  Table  18  were  calculated  with  the  assumption  that  trials  eanceled 
because  of  ADS  system  failures  resulted  in  a  loss  of  7  hours  of  test  time.  This  figure  was  used 
because  each  ETE  Test  vignette  was  designed  to  support  7  hours  of  testing  with  more  or  less 
time  possible.  ADS  system  failures  that  did  not  result  in  a  cancellation  of  the  trial  (i.e.,  failures 
that  did  not  result  in  a  breakdown  of  the  ETE  Test  loop)  were  not  included.  The  table  shows  that 
29  hours  of  test  time,  the  equivalent  of  more  than  4  days  of  testing,  were  lost  because  of  ADS 
system  failures,  with  the  large  majority  of  this  time  lost  because  of  problems  with  the  VSTARS 
software  at  Northrop  Grumman. 

Table  18  shows  the  lost  time  for  both  the  SIT  LSP  and  LFP.  The  LSP  was  the  only  phase  that 
experienced  lost  test  time  because  of  ADS  system  failures.  The  15  percent  lost  time  figure  for 
LSP  was  nearly  evenly  split  between  hardware  and  software  failures.  Most  of  the  software 
failures  required  only  minutes  while  simulation  or  network  interfaces  were  reset.  The  three 
hardware  failures  included  a  missile  hardware  failure  for  approximately  one  hour  and  two 
failures  involving  cooling  or  power  supplied  to  a  node.  During  the  LFP  there  were  three 
hardware  failures,  none  of  which  resulted  in  lost  test  time. 

Personnel  and  Procedures 

For  all  the  JADS  tests,  detailed  problem  logs  were  kept  at  each  of  the  sites  to  provide  insight  into 
the  impacts  of  different  types  of  ADS  system  RM&A  problems.  JADS  personnel  used  these  logs 
to  document  the  behavior  of  existing  facility  hardware  and  software,  hardware  and  software 
implemented  to  enable  distribution,  communications  equipment,  and  the  distributed  network 
architecture.  In  addition,  EW  Test  personnel  used  these  problem  logs  to  document  personnel  and 
procedural  problems  encountered  during  testing.  Interestingly,  for  both  Phases  2  and  3  of  the 
EW  Test,  personnel  and  procedural  problems  accounted  for  the  seeond  largest  number  of  delays 
and  test  trials  lost,  next  to  the  combined  ADS  system  hardware  and  software  problems. 


65 


Table  19  shows  the  impact,  in  terms  of  lost  test  trials,  of  personnel  and  procedure  problems 
experienced  during  the  EW  Test. 

Table  19.  EW  Test  Trials  Canceled/Aborted  Because  of  Personnel  and  Procedural 

Problems 


EW 

Test 

Phase 

Total  # 
Of  Trials 

Total  #  of 
Aborted 
Trials 

#  of  Trials  Lost 
Because  of  Personnel 
and 

Procedure  Problems 

%  of  Trials  Lost 
Because  of  Personnel 
and 

Procedure  Problems 

2 

363  ^ 

95 

10 

3% 

3 

270 

32 

10 

4% 

Phase  2  personnel  and  procedural  difficulties  included  missing  script  files,  incorrect  script  input, 
miscommunication  among  test  controllers  and  test  station  operators,  operator  tardiness,  and 
operator  slowness  or  unfamiliarity  with  procedures.  As  a  result  of  these  problems,  several 
procedures  and  checklists  were  made  stricter  for  Phase  3.  Following  these  changes,  difficulty 
with  personnel  and  procedures  during  Phase  3  execution  consisted  almost  entirely  of  operator 
error  based  on  miscommunication  or  simple  lack  of  concentration. 

Neither  the  ETE  Test  nor  the  SIT  was  affected  as  much  by  personnel  and  procedural  issues  as  the 
EW  Test.  The  ETE  Test  trials  lasted  an  average  of  7  hours.  If  personnel  or  procedural  problems 
occurred,  these  problems  could  be  resolved  without  any  impact  to  the  trials.  For  this  reason,  any 
personnel  or  procedural  problems  were  minor  and  were  not  analyzed.  For  the  SIT,  very  few 
personnel  had  to  interact  during  the  trial  runs  eliminating  the  potential  for  nearly  any  type  of 
personnel  failure.  The  most  critical  interaction  was  between  the  two  pilots  performing  the 
desired  flight  engagements.  In  addition,  the  SIT  had  more  extensive  rehearsal  time  to  refine 
procedures  and  train  operators,  resulting  in  the  reduction  of  potential  procedural  failures.  As  was 
the  case  for  the  ETE  Test,  personnel  and  procedural  failures  were  not  a  major  issue  for  the  SIT, 
and  data  on  these  failures  were  not  analyzed. 

Network  RM&A 

The  same  logs  and  analysis  procedures  used  to  study  ADS  hardware  and  software  RM&A  were 
also  applied  to  network  evaluation  for  each  of  the  three  JADS  tests.  Again,  data  collectors 
annotated  all  problems  encountered,  as  well  as  their  causes,  and  test  controllers  documented  the 
overall  status  of  the  network  and  test  trials.  In  addition,  commercial  and  in-house  network 
monitoring  tools  (e.g.,  SPECTRUM,  NETVisualizer™,  JADS  Link-Availability  Monitor)  were 
used  to  monitor  the  status  of  all  network  equipment  and  links  among  nodes.  Any  problems 
detected  by  these  monitoring  tools  were  brought  to  the  test  controller’s  attention  via  on-screen 
lights  or  sounds,  as  well  as  stored  to  databases  for  further  detailed  analysis.  The  results  of  JADS 
network  equipment  self-diagnostics  were  documented  via  line  printers  in  terms  of  a  brief 
explanation  of  the  problem,  the  time,  and  the  link(s)  involved. 
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The  following  paragraphs  and  tables  detail  the  results  of  network  reliability  for  each  of  the  three 
tests  and  the  impacts  to  testing  of  that  reliability. 

Table  20.  ETE  Test  Cancellations/Postponements  Because  of  Network  Outages 


Phase 

Affected  Trials 

Comments 

2 

2 

T-1  outage  due  to 
contractor/hurricane 

3 

N/A 

IHIBIIillljl 

4 

N/A 

Table  20  shows  each  ETE  Test  canceled/postponed  trial.  Overall,  the  ETE  Test  network 
performed  with  high  reliability.  No  trials  were  canceled  or  not  used  because  of  network 
problems.  However,  two  Phase  2  trials  were  postponed  when  the  agency  contracted  to  provide 
network  service  inadvertently  terminated  use  of  the  T-1  line  at  the  Northrop  Grumman  node. 
The  coincidence  of  a  hurricane  hitting  the  Florida  and  Gulf  coasts  during  the  same  time  period 
delayed  the  reactivation  of  this  link.  As  a  result,  the  test  was  extended  two  days  after  the 
scheduled  test  period. 


Table  21.  ETE  Test  Network  Downtime 


ETE  Test 

Phase 

Time 

Scheduled  for 
Testing 

Time 
Network 
Unavailable 
for  Testing 

%  of  Time 
Network 
Unavailable 

2 

35  hrs,  27 
mins 

8  mins 

0.38% 

3 

14  hrs,  13 
mins 

1 1  mins 

1.29% 

4 

50  hrs,  9  mins 

39  mins 

1.30% 

Total 

99  hrs,  49 
mins 

58  mins 

0.97% 

The  ETE  Test  network  downtime  experienced  during  test  trials,  by  phase  and  for  the  test  as  a 
whole,  can  be  seen  in  Table  21.  Aside  from  the  postponed  trials  caused  by  the  contract  agency 
error,  the  network  was  unavailable  for  less  than  one  hour  of  the  ETE  Test  over  the  course  of  all 
three  phases.  The  majority  of  this  downtime  was  due  to  network  routers  at  the  distributed  nodes 
failing  momentarily.  This  suggests  a  very  high  level  of  network  reliability. 
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Table  22.  EW  Test  Network  Downtime 


EW  Test 

Phase 

Time 

Scheduled  for 
Testing 

Time 
Network 
Unavailable 
for  Testing 

%  of  Time 
Network 
Unavailable 

2 

57  hrs,  12 
mins 

17  mins 

<  0.50% 

3 

48  hrs,  48 
mins 

1  mins 

<  .03% 

Total 

106  hrs 

1 8  mins 

<  .28% 

Table  22  shows  the  EW  Test  network  downtime.  Network  problems  encountered  during  the  EW 
Test  Phase  2  were  limited,  resulting  in  just  over  17  minutes  of  lost  test  time  and  two  aborted 
runs.  Specific  problems  included  a  bad  Integrated  Digital  Network  Exchange  (IDNX''^*^)  voice 
card  at  ACETEF,  ACETEF  router  problems,  and  a  link  between  ACETEF  and  AFEWES  going 
down  momentarily.  The  bad  voice  card  resulted  in  two  additional  trials  being  run  using  non- 
secure  voice  communication.  During  Phase  3,  network  problems  caused  just  one  trial  to  be 
aborted.  The  JADS-AFEWES  link  and  a  minor  crypto  problem  were  the  cause  of  this  brief 
network  downtime.  The  impact  of  this  downtime  was  insignificant  with  a  minimal  loss  of  TSPI 
data  at  AFEWES  and  two-way  losses  of  best-effort  (UDP)  data  and  a  delay  of  reliable  (TCP) 
traffic  over  one  network  link. 


Table  23.  SIT  Network  Downtime 


srr 

Phase 

Time 

Scheduled  for 
Testing 

Time 
Network 
Unavailable 
for  Testing 

%  of  Time 
Network 
Unavailable 

LSP 

19  hrs,  42 
mins 

0  mins 

0% 

LFP 

3  hrs,  39  mins 

0  mins 

0% 

Total 

23  hrs,  21 
mins 

0  mins 

0% 

Table  23  shows  that  there  were  no  network  outages  during  either  phase  of  the  SIT.  The  average 
length  of  a  daily  test  using  the  network  was  approximately  five  hours.  Since  there  were  no 
outages,  there  was  no  impact  on  the  test  because  of  network  reliability. 
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2.2.2  Critical  Constraints  and  Concerns  in  ADS  Support  Systems  for  T&E 
2.2.2.1  Data  Management  and  Analysis  Systems 

The  three  JADS  tests  showed  that  data  collection,  retrieval,  and  storage  methodologies  do  not 
need  to  be  more  complex  for  distributed  tests  than  for  traditional  non-ADS  tests,  and  that,  in 
some  ways,  ADS  data  management  techniques  can  be  simpler  and  more  efficient  than  their 
traditional  counterparts.  JADS  tests  showed  that  both  automated  and  handwritten  data  were 
easily  collected  and  retrieved  from  distributed  sites  with  electronic  data  transfer  via  file  transfer 
protocol  (FTP)  speeding  the  process  and  allowing  for  more  timely  and  thorough  initial  review. 

The  test  teams  used  many  similar  data  collection  and  retrieval  methods  with  data  handling 
dependent  primarily  on  the  particular  format  of  the  data  generated,  either  electronic  (produced 
and  collected  by  automated  tools  and  systems)  or  operational  (usually  in  the  form  of  handwritten 
or  typed  log  forms).  For  the  ETE  Test  and  the  SIT,  automated  data  were  collected  through  the 
use  of  tools  such  as  the  JADS  logger,  which  collected  PDU  log  files,  and  a  SPECTRUM  logger, 
which  monitored  network  performance.  These  data  were  retrieved  at  the  TCAC  following  each 
test  day  using  FTP.  Operational  data  were  collected  using  log  sheets  at  each  test  node.  JADS 
personnel  transported  these  data  to  Albuquerque  following  each  test  phase. 

Data  handling  for  the  EW  Test  differed  little,  although  data  from  a  few  additional  automated  data 
collection  tools  (e.g.,  ETHERPEEK'^m  data  packet  sniffers  implemented  across  the  network  to 
characterize  data  transfer),  as  well  as  distributed  simulation  (e.g.,  AFEWES  threat  system) 
performance  data,  added  to  the  complexity  and  amount  of  data.  In  addition,  the  JADS  EW  Test 
log  files,  generated  at  each  site  and  collected  daily,  handled  HLA-compliant  federation  data 
versus  the  ETE  Test’s  DIS  PDUs.  The  test  teams  encountered  no  significant  problems  during  the 
collection  and  retrieval  efforts  for  either  the  operational  or  automated  data. 

Similarly,  no  major  problems  were  experienced  in  data  storage.  All  log  files  from  each  test  node 
were  stored  on  hard  drives  and  backed  up  to  tape.  The  only  adjustment  required  for  data  storage 
for  the  ETE  Test  was  the  installation  of  4-gigabyte  (GB)  hard  drives  because  of  the  large  size  of 
the  log  files.  The  EW  Test  team  stressed  the  importance  of  having  good  documentation  for 
storage  of  electronic  data  including  clear  and  detailed  directory  structures.  Handwritten  log 
sheets  and  operator  questionnaires  from  each  node  were  maintained  in  test  controller  or  analyst 
notebooks  for  easy  access.  The  successful  implementation  of  these  storage  methods,  along  with 
the  collection  and  retrieval  methods  employed,  show  the  level  of  simplicity  which  can  be 
maintained  for  data  management  of  ADS-enhanced  tests. 

The  EW  Test,  with  its  ADS  and  non-ADS  test  phases,  enabled  analysts  to  make  comparisons 
between  the  data  management  techniques  used.  During  the  OAR  test  (non-ADS),  personnel  at 
the  threat  sites  and  in  the  control  center  performed  written  data  collection.  Digital  range  data 
were  delivered  post-mission  to  JADS  where  they  were  catalogued,  analyzed,  and  stored.  This 
data  collection  method  did  not  allow  for  a  great  deal  of  immediate  feedback  on  the  SUT  and 
threat  performance.  On  the  other  hand,  the  delay  between  missions  did  allow  for  fixes  in  data 
collection  processes,  data  collection  equipment,  and  the  SUT  itself.  For  Phase  2  and  3  (ADS), 
written  data  were  collected  by  personnel  at  each  active  threat,  by  the  controller  at  each  site,  and 
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in  the  TCAC  by  operators  performing  real-time  analysis.  This  data  collection  method  allowed 
for  more  immediate  feedback  as  to  how  all  federation  systems  were  performing  and  provided 
insight  into  the  quality  of  the  SUT  data  being  generated,  allowing  system  flaws  to  be  fixed 
almost  immediately.  While  the  ADS-enhanced  tests  required  more  personnel  during  test 
execution  to  perform  data  collection  and  retrieval,  the  result  was  less  delay  in  acquiring  in-depth 
knowledge  about  test  performance. 

Another  difference  in  data  management  between  AE)S  testing  and  non-ADS  testing  for  the  EW 
Test  was  the  technique  by  which  electronic  data  were  transferred.  The  OAR  test  allowed  for  a 
single  compact  disk  to  be  delivered  to  JADS  containing  all  the  necessary  data,  whereas  the  ADS 
tests  used  daily  FTP  transfers  to  accomplish  most  of  the  data  delivery.  The  OAR  approach 
packaged  the  data  nicely  and  made  them  easier  to  manage  but  also  required  a  long  time  to 
discover  and  replace  any  corrupted  or  missing  data.  The  distributed  tests  allowed  for  quicker 
responses  to  data  being  lost  or  not  transferred  but  also  required  much  more  management  and 
controlled  placement  of  the  information. 

Lastly,  there  were  differences  in  the  actual  amount  and  type  of  data  created  by  the  two  types  of 
tests.  ADS-enhanced  tests  produce  more  data  than  non-ADS  tests.  In  addition  to  the 
requirement  to  collect  SUT,  threat,  and  operator  performance  data,  network  data  must  also  be 
collected.  Overall,  ADS-enhanced  testing  added  more  flexibility  for  data  management  and 
quicker  insight  into  the  quality  of  collected  test  data,  but  it  did  require  more  organization  and 
documentation  to  make  data  handling  techniques  effective. 

2.2.2.2  Configuration  Management 

Configuration  management  for  the  JADS  tests  proved  to  be  a  difficult  task.  ADS-enhanced 
testing  makes  configuration  management  more  complex  than  in  single-site  test  efforts  because  of 
the  different  organizations  involved.  The  JADS  test  managers  did  not  have  complete  control 
over  the  software  or  systems  at  the  distributed  test  nodes,  in  part  due  to  limited  budget  and  low 
priority.  Existing  configuration  management  procedures  at  the  different  facilities  were  used. 
This  alleviated  some  test  problems,  but  since  each  facility  possessed  individual  configuration 
management  processes,  a  single  configuration  management  standard  could  not  be  created.  This 
is  a  potential  weakness  for  ADS-enhanced  tests  in  general  but  not  a  disastrous  one.  This  lack  of 
centralized  control  was  overcome  by  the  JADS  tests  in  a  variety  of  ways. 

JADS  test  managers  maintained  close  contact  with  personnel  from  contracted  agencies  and 
conducted  frequent  meetings  with  test  participants.  This  gave  the  test  manager  the  ability  to 
track  configuration  management  progress  and  problems  and  provided  a  forum  for  system  experts 
to  resolve  issues.  For  ADS-enhanced  tests  in  general,  this  increased  level  of  management  is  vital 
to  ensure  effective  configuration  management  with  a  focus  on  top-down  management  as  a 
necessity.  Frequent  integrated  product  teams  (IPTs),  detailed  documentation  and  formalized 
procedures  should  also  be  maintained  to  ensure  successful  configuration  management. 

If  a  stringent  configuration  management  plan  cannot  be  adhered  to  by  all  involved  test  sites,  then 
the  use  of  integration  testing  is  critical  in  ensuring  effective  test  management  and  execution. 
This  was  the  case  for  the  EW  and  ETE  tests.  For  the  ETE  Test,  exhaustive  integration  testing 


70 


was  conducted  to  guarantee  all  test  hardware,  software,  and  interfaces  could  be  used  in  their 
current  configuration  with  modifications  performed  as  necessary.  For  the  EW  Test,  intensive 
integration  testing  was  performed  on  all  software  before  runs  for  record  began.  This  was 
especially  important  because  of  software  changes  made  at  multiple  test  sites  prior  to  the  Phase  2 
and  Phase  3  test  events.  The  problem  with  this  approach  is  the  increased  need  for  software 
developers  who  can  change  software  very  quickly  if  problems  are  encountered.  This  puts  a 
heavy  responsibility  on  the  software  developers  who  may  not  be  able  to  support  such  activities. 

Another  key  point  to  remember  when  conducting  integration  testing  for  an  ADS-enhanced  test  is 
to  test  with  the  entire  architecture.  During  the  SIT,  a  software  change  was  made  to  one  node  and 
tested  in  a  stand-alone  manner.  Although  the  fix  appeared  to  work,  when  the  node  was  tested 
with  the  entire  network,  the  problem  still  existed.  Just  as  configuration  management  is  critical, 
robust  integration  testing  with  the  entire  network  is  a  must. 

2.2.3  Methodologies  Associated  with  ADS  for  T&E 

This  section  outlines  useful  procedures  in  implementing  ADS  based  on  steps  given  in  the  HLA 
Federation  Development  and  Execution  Process  (FEDEP)  model^.  A  more  detailed  version  of 
this  checklist  is  available  in  A  Test  Planning  Methodology  -  From  Concept  Development 
Through  Test  Execution^ 

STEP  1 :  Define  Distributed  Test  Objectives 
Activity  1.1;  Identify  Needs 

-  Activity  Purpose:  develop  clear  understanding  of  the  problem  to  be  addressed  by  the 
distributed  test 

-  Activity  Inputs:  program  objectives  and  information  on  resources  available  to  support  a 
distributed  test 

-  Activity  Output:  needs  statement 
Activity  1 .2:  Develop  Objectives 

-  Activity  Purpose:  refine  the  needs  statement  into  a  more  detailed  set  of  specific 
objectives  for  the  distributed  test 

Activity  Inputs:  needs  statement  from  previous  activity 

Activity  Outputs:  statement  of  the  test  objectives  and  initial  planning  documents 

STEP  2:  Develop  Conceptual  Model 
Activity  2. 1 :  Develop  Scenario 

Activity  Purpose:  develop  a  functional  specification  of  the  test  scenario 

-  Activity  Inputs;  operational  context  constraints  specified  in  the  test  objective  statement 
Activity  Output:  test  scenario  description 

Activity  2.2:  Perform  Conceptual  Analysis 

“High  Level  Architecture  Federation  Development  and  Execution  Process  (FEDEP)  Model, 
Version  1.4,”  9  June  1999  available  from  the  Defense  Modeling  and  Simulation  Organization 
(DMSO)  HLA  web  site  located  at  http://hla.dmso.mil/. 

’  Available  at  http://www.jads.abq.com.  After  1  March  2001  refer  requests  to  the  Joint  Program 
Office  Technical  Library,  2001  North  Beauregard  St.  Suite  800,  Alexandria,  Virginia  22311. 
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Activity  Purpose:  produce  a  conceptual  model  of  the  distributed  testing  environment 

-  Activity  Inputs:  test  scenario  description  from  previous  activity,  test  objectives  statement, 
and  any  doctrine  and  tactics  appropriate  for  the  scenario 

Activity  Output:  conceptual  model  that  is  a  description  of  the  players,  their  actions,  and 
any  interactions  among  players  that  need  to  be  included  in  the  distributed  test  in  order  to 
achieve  all  test  objectives 

Activity  2.3:  Develop  Distributed  Test  Requirements 

Activity  Purpose:  define  top-level  test  requirements 
Activity  Input:  conceptual  model  from  previous  activity 

Activity  Output:  (1)  requirements  based  on  the  distributed  test  objectives  are  directly 
testable  and  provide  the  implementation  level  guidance  needed  to  design  and  develop  the 
distributed  test,  and  (2)  requirements-based  criteria  for  evaluating  test  results.  Major  top- 
level  requirements  that  should  be  addressed  include  the  following 

-  Fidelity  requirements  for  all  players  represented  in  the  test  scenarios 

—  Interaction  requirements  specifying  information  types  that  must  be  exchanged  among 
players  to  permit  interactions 

—  Latency  requirements  for  the  maximum  acceptable  latency  for  each  pair  of  interacting 
players 

-  Data  reliability  requirements  specifying  the  maximum  acceptable  level  of  distributed 
testing-induced  errors,  such  as  dropout  rate  and  out-of-order  messages 

—  Data  analysis  requirements  including  a  data  management  and  analysis  plan  (DMAP) 
detailing  the  analysis  approach  for  each  test  objective 

STEP  3:  Design  Distributed  Test 
Activity  3.1:  Select  Participants 

Activity  Purpose:  determine  the  suitability  of  individual  player  representations  (e.g., 
simulations,  HWIL  labs,  or  live  players/ranges)  to  become  participants  in  the  distributed 
test 

Activity  Input:  conceptual  model  developed  in  Activity  2.2 

-  Activity  Output:  identification  of  the  specific  player  representations  selected 
Activity  3.2:  Allocate  Functionality 

Activity  Purpose:  allocate  the  responsibility  to  represent  the  entities  and  actions  in  the 
conceptual  model  to  the  participants 

Activity  Inputs:  identify  participants  from  the  previous  activity  along  with  the  test 
requirements,  the  test  scenario,  and  the  conceptual  model 

Activity  Output:  allocate  requirements  for  the  participants  including  any  requirements  for 
modifying  existing  player  representations  or  designing  new  ones.  The  steps  for  this 
activity  are  as  follows 

—  Develop  allocated  requirements  including  the  following 

—  Data  requirements  including  data  rates,  TSPI  accuracy  and  smoothness,  data  time 
stamp  accuracy,  classification  of  the  data  and  any  security  handling  procedures 
—  Data  synchronization  requirements 
—  Real-time  data  processing  requirements 
—  Data  collection/instrumentation  requirements 
—  Determine  if  modifications  to  the  selected  player  representations  are  needed,  such  as 
—  Simulation  modifications  needed  to  utilize  external  inputs 
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—  Simulation  modifications  needed  to  generate  the  required  outputs 
—  Data  processing  modifications  needed  to  meet  TSPI  accuracy,  smoothness,  and 
latency  requirements 

—  Facility  modifications  needed  for  a  replay  capability  that  can  be  used  during 
integration  testing 
Activity  3.3:  Prepare  Plan 

Activity  Purpose:  develop  a  coordinated  plan  to  guide  the  development,  test,  and 
execution  of  the  distributed  test 

Activity  Inputs:  initial  planning  documents  prepared  during  the  development  of  the  test 
objectives  (Activity  1 .2)  and  the  allocated  participant  requirements 

-  Activity  Outputs:  detailed  planning  documents  including  a  detailed  test  plan  and  DMAP; 
a  verification,  validation  and  accreditation  (VV&A)  plan;  and  an  interface  control 
document  (ICD) 

STEP  4:  Develop  Distributed  Test 

Activity  4.1:  Develop  federation  object  model  (FOM) 

-  Activity  Purpose:  develop  the  FOM  (if  HLA  is  to  be  implemented) 

-  Activity  Inputs:  detailed  planning  documents  and  the  allocated  participant  requirements 

-  Activity  Outputs:  FOM  and  federation  execution  data  (FED)  file,  if  appropriate 
Activity  4.2:  Establish  Participant  Agreements 

Activity  Purpose:  establish  all  agreements  among  participants  necessary  for  a  fully 
consistent,  interoperable,  distributed  simulation  environment 

Activity  Inputs:  test  scenario,  conceptual  model,  and  FOM  (if  HLA  is  to  be  implemented) 
Activity  Output:  revised  participant  allocated  requirements  including  any  requirements 
for  additional  modifications.  The  steps  for  this  activity  are 
—  The  participant  interaction  requirements  are  finalized  including  the  following 
—  Determine  the  data  protocols  to  be  used 

—  Interface  requirements  including  those  for  simulation  interfaces,  special-purpose 
interfaces,  and  interfaces  to  the  RTI  for  HLA  implementation 
—  Operational  issues  and  policies  are  also  addressed  and  resolved  including  the 
following 

—  Terrain  database  requirements 
—  Post-test  data  management  requirements 
—  Test  control  and  monitoring  requirements 
Activity  4.3:  Implement  Participant  Modifications 

Activity  Purpose:  implement  participant  modifications  identified  in  previous  activities 
Activity  Input:  updated  allocated  participant  requirements 

-  Activity  Outputs:  modified  participants 

STEP  5:  Integrate  and  Test  Architecture 
Activity  5.1:  Plan  Execution 

Activity  Purpose:  define  and  develop  the  full  set  of  information  required  to  support  the 
distributed  test  execution 

-  Activity  Inputs:  FOM  and  FED  file  (if  HLA  is  to  be  implemented),  test  scenario,  and 
detailed  test  plan 
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-  Activity  Outputs:  refined  and  detailed  integration  test  plan,  VV&A  plan,  and  test 
procedures.  The  steps  for  this  activity  are 
—  Complete  detailed  network  design  including  the  following 

_ Determine  if  data  from  each  node  will  be  broadcast,  multicast,  or  unicast 

(transmitted  point-to-point) 

_ Determine  if  data  are  to  be  transmitted  using  best  effort  or  reliable  procedures 

—  Determine  network  security  approach  to  be  implemented 
—  Determine  the  wide  area  network  (WAN)  bandwidth  requirement 
—  Conduct  surveys  of  each  site  to  be  linked  by  the  network 

-  Refine  the  integration  test  plan  to  include  step-by-step  systematic  integration  testing 

procedures  that  address  the  following  ... 

—  Procedures  for  verifying  any  simulation/range  facility  modifications  m  a 

systematic  stand-alone  fashion 

—  Procedures  for  initially  testing  each  WAN  link  separately 

-  Procedures  for  testing  each  simulation-to-simulation  connection  with  all  network 
nodes  connected 

-  Procedures  for  testing  the  voice  communications  with  all  equipment  and 
personnel  as  in  the  actual  test 

—  Develop  detailed  test  control  procedures  and  a  security  test  and  evaluation  plan 

Activity  5.2:  Integrate  and  Test  Distributed  Architecture 

-  Activity  Purpose:  bring  all  the  distributed  test  participants  into  a  unifying  operating 
environment  and  verify  that  they  can  all  interoperate  to  the  degree  required  to  achieve 
the  test  objectives 

Activity  Inputs:  detailed  test  plan,  VV&A  plan,  and  test  procedures 

-  Activity  Outputs:  refined  test  procedures,  VV&A  results,  and  a  distributed  testing 
architecture  which  has  been  thoroughly  tested  and  is  ready  for  test  execution.  Key 
steps  during  this  activity 

—  Install  the  distributed  test  network 

—  Select  and  procure  the  WAN  to  be  used  to  link  the  facilities 
—  Select,  procure,  and  install  the  network  hardware  to  be  used  including  routers, 
channel  service  units  (CSUs)/data  service  units  (DSUs),  multiplexers, 

encryptors,  etc.  ■  u  u 

—  Build/procure  the  interfaces  necessary  for  linking  in  accordance  with  the 

requirements  developed  during  Activity  4.2 
—  Select,  procure,  and  install  test  control  hardware  and  software  based  on  the 
requirements  developed  in  Activity  3.1 

_ Select,  procure  and/or  develop,  and  install  network  analysis/monitoring  tools 

-  Key  testing  steps  during  this  activity 

—  Perform  compliance  testing,  as  specified  in  the  VV&A  plan 
—  Perform  integration  testing,  as  specified  in  the  integration  test  plan 
—  Perform  risk  reduction  missions 
—  Perform  validation,  as  specified  in  the  VV&A  plan 
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STEP  6:  Execute  Distributed  Test  and  Analyze  Results 
Activity  6.1 :  Execute  Distributed  Test 

-  Activity  Purpose:  exercise  all  distributed  test  participants  as  an  integrated  whole  to 
generate  required  outputs  and  achieve  the  stated  test  objectives 

-  Activity  Inputs:  refined  test  procedures  and  tested  distributed  testing  architecture 
from  integration  testing 

-  Activity  Output:  raw  test  results.  The  following  considerations  apply  during 
execution 

—  Pre-  and  post-test  briefings  are  essential 

—  Deploy  test  force  personnel  to  the  range/sites  several  hours  prior  to  the  mission  to 
confirm  that  the  preparations  have  been  completed 
—  Centralized  test  control/management  and  execution  monitoring  must  be 
maintained,  following  detailed  test  control  procedures  and  checklists 
—  Data  collected  during  execution  are  used  to  support  both  quick-look  and  detailed 
post-test  analyses 

—  Strict  attention  must  be  given  to  maintaining  the  security  posture  of  the  distributed 
testing  architecture  during  execution 

—  Conduct  an  after  action  review  immediately  after  the  test  in  order  to  gather 
important  information  from  each  facility,  to  formulate  a  course  of  action  for 
correcting  any  problems,  and  to  prepare  for  the  next  period  of  testing 
Activity  6.2:  Process  Output 

-  Activity  Purpose:  post-process  (as  necessary)  the  output  collected  during  test 
execution 

-  Activity  Input:  raw  test  results  from  test  execution 

-  Activity  Output:  derived  test  results 
Activity  6.3:  Prepare  Results 

Activity  Purpose:  (1)  evaluate  the  data  analysis  results  in  order  to  determine  if  all  test 
objectives  have  been  met  and  (2)  identify  legacy  products  and  make  them  available  to 
other  programs 

Activity  Input:  derived  test  results,  along  with  the  test  evaluation  criteria  from 
Activity  2.3 

-  Activity  Output:  documented  test  results  and  legacy  products 
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2.3  General  Requirements 

Issue  three  of  the  JADS  charter  required  that  the  JTF  identify  requirements  that  must  be 
introduced  into  ADS  systems  if  they  are  to  support  a  more  complete  T&E  capability  m  the 
future. 


Issue 

Objective 

Issue  3:  What  are  the  requirements  that 
must  be  introduced  into  ADS  systems  if 
they  are  to  support  a  more  complete  T&E 
capability  in  the  future? 

Objective  3-1:  Identify  requirements 
for  ADS  systems  that  would  provide 
a  more  complete  T&E  capability  in 
the  future. 

The  general  requirements  that  are  necessary  if  ADS  systems  are  to  support  a  more  complete 
T«feE  capability  in  the  future  include  standards  for  network  interoperability  and  multilevel 
security,  network  performance  requirements,  expertise  to  support  distributed  tesdng  and 
leadership.  Sections  2.3. 1-4  below  discuss  each  of  these  general  requirements  and  section  2.3.5 
provides  specific  recommendations. 

2.3.1  Standards 

DMSO,  the  Simulation  Interoperability  Standards  Organization  (SISO),  Foundation  Initiative 
2010,  and  many  other  government  and  commercial  organizations  are  developing  open  systems 
standards  for  technical  interoperability  among  live,  virtual,  and  constructive  siniulations. 
Operational  system  interoperability  standards  are  also  being  developed,  and  the  distinction 
between  operational  interoperability  standards  and  simulation  interoperability  standards  is  very 
slight  in  many  cases.  Multilevel  security  standards  and  distributed  test  control  standards  are 
areas  which  require  additional  attention.  Although  technical  standards  have  a  long  way  to  go,  an 
adequate  process  appears  to  be  in  place  to  develop  them.  A  similar  process  needs  to  be  put  in 
place  to  develop  programmatic  standards  that  will  support  distributed  testing  across  all  phases  of 
the  acquisition  process.  As  in  the  technical  standards,  these  programmatic  standards  need  to  also 
support  international  distributed  testing  and  training. 

2.3.2  Networking 

Network  requirements  (i.e.,  expected  data  rate,  latency  budget,  communications  protocols, 
control  and  management  of  the  network)  need  to  be  defined  early  in  the  process.  These 
requirements  must  be  clearly  defined  and  forwarded  to  the  Defense  Information  Systems  Agency 
(DISA)  for  evaluation  of  how  they  can  best  support  the  requirements,  either  through  DISA 
common  user  networks  (i.e..  Defense  Simulation  Internet  (DSI),  Secret  Internet  Protocol  Router 
Network  (SIPRNET),  Defense  Research  and  Engineering  Network  (DREN),  etc.),  or  by  granting 
a  waiver  exempting  use  of  common  user  networks  in  order  to  build  a  private  network  suitable  for 
the  requirements.  It  should  be  noted  that  if  DISA’s  common  user  networks  are  used,  DISA  will 
allow  connection  to  only  one  of  the  networks.  These  networks  cannot  be  interconnected  because 
of  security,  interoperability,  and  tariff  constraints.  Thus,  it  requires  close  coordination  with 
DISA  to  ensure  the  network  of  choice  will  support  all  requirements. 
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Time  is  of  the  essence.  Careful  planning  and  consideration  must  be  given  to  the  schedule  for 
implementing  the  communications  network.  On  average,  once  the  requirements  are  defined  and 
a  networking  solution  is  decided  upon,  it  will  take  a  minimum  of  120  days  (if  a  DISA  common 
user  network  is  to  be  utilized,  it  could  take  more  than  180  days)  to  procure  and  install  the 
necessary  communications  circuits  and  networking  hardware.  Also,  it  will  take  a  minimum  of  90 
days  to  obtain  the  necessary  communications  security  (COMSEC)  equipment  and  keying 
material  to  encrypt  the  data.  It  may  take  longer  if  a  COMSEC  subaccount  needs  to  be 
established.  In  addition,  time  must  be  allocated  in  the  schedule  to  install,  test,  and  validate 
communications  network  performance.  On  average,  JADS  allocated  one  week  per  site  to 
accomplish  these  tasks. 

2.3.3  Expertise 

Distributing  testing  requires  expertise  in  many  disciplines,  and  the  JADS  experience  has  been 
that  very  few  organizations  have  the  required  depth  and  breadth  of  expertise  required  to 
efficiently  conduct  a  successful  distributed  test.  JADS  had  to  develop  a  combination  of 
government  and  contractor  expertise  to  support  each  of  the  JADS  tests.  In  doing  so,  centers  of 
distributed  testing  expertise  were  established,  but  both  the  government  and  contractor  expertise 
is  very  perishable.  There  is  a  critical  requirement  to  establish  persistent  centers  of  excellence 
within  OSD  and  the  services  to  support  distributed  testing,  or  we  will  be  required  to  reinvent  the 
proverbial  wheel  many  more  times. 

2.3.4  Leadership  Support 

Continued  leadership  support  is  required  to  facilitate  the  use  of  distributed  testing  as  a  widely 
accepted  and  used  test  tool.  In  particular,  middle  management  needs  to  be  strongly  encouraged 
to  seriously  consider  the  use  of  distributed  testing  to  support  future  complex  acquisition 
programs.  The  previously  listed  requirements  are  areas  that  require  immediate  and  sustained 
leadership  support. 

2.3.5  Recommendations 

The  following  list  of  recommendations  summarizes  the  requirements  for  ADS  systems  that  are 
necessary  to  provide  a  more  complete  T&E  capability  in  the  future. 

-  Address  ADS  approaches  in  the  test  and  evaluation  master  plan  (TEMP). 

-  Focus  on  the  ability  of  ADS  to  overcome  any  identified  test  limitations. 

-  Program  managers  and  operational  test  agencies  (OTAs)  should  embrace  and  implement 
STEP  and  SBA.  ADS  is  an  enabling  technology  for  STEP  and  SBA. 

-  Use  the  IPT/integrated  product  and  process  development  (IPPD)  process  to  facilitate 
utilization  of  assets  across  phases  of  development  including  requirements  definition, 
engineering,  manufacture  and  development,  test  and  evaluation,  operations  and  training. 
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-  Mid-  and  upper-management  should  encourage/require  T&E  vision  beyond  the  scope  of 
individual  test  events  and  specific  systems. 

-  When  making  an  ADS  go/no  go  decision,  compare  ADS-enhanced  testing  costs  to  the  costs 
of  the  alternative  method(s). 

-  Use  JADS-developed  ADS  cost  guidance  to  helpjdentify  the  optimal  mix  of  ADS  enhanced 
testing  and  traditional  means  of  testing. 

-  DoD  should  develop  infrastructure  to  reduce  the  costs  of  linking. 

-  Use  an  ADS  test  environment  over  the  life  of  a  program. 

-  Incorporate  ADS  into  the  curricula  of  formal  T&E  and  acquisition  schools. 

-  DoD  should  nurture  groundbreaking  programs,  such  as  Foundation  Initiative  2010,  Joint 
Strike  Fighter. 

-  Use  JADS-developed  ADS  methodologies  such  as  test  planning,  verification  and  validation. 

-  Each  organization  using  ADS  should  plan  for  a  centralized  test  control  and  analysis 
capability.  Such  a  facility  can  be  low  cost  and  located  anywhere.  In  addition  to  test  control, 
it  can  be  used  to  enhance  real-time  data  analysis  and  test  efficiency. 
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3.0  Lessons  Learned 


Near  the  end  of  the  JADS  program,  we  came  to  the  realization  that  the  term  “Advanced 
Distributed  Simulation  (ADS)”  was  potentially  misleading  when  used  in  the  context  of  T&E. 

The  word  “simulation”  is  something  of  a  turn  off  for  testers,  because  they  are  sharply  focused  on 
collecting  real  data  whenever  they  can.  In  fact,  distributed  architectures  can  be  used  effectively 
to  support  actual  testing.  A  SUT  can  be  incorporated  into  a  distributed  architecture  and 
subjected  to  valid  stimuli  that  can  originate  at  a  number  of  remote  sites.  The  collection  of  stimuli 
may  indeed  create  an  “artificial  environment”  which  is  perceived  by  the  SUT,  but  if  the  stimuli 
are  valid,  real  performance  data  can  be  collected  on  the  performance  of  the  SUT.  That  is  testing. 
(As  an  aside,  even  in  traditional  testing,  the  test  environment  is,  more  often  than  not,  artificial.) 
The  fact  that  a  stimulus  originates  at  some  distance  from  the  SUT  location  is  immaterial  so  long 
as  it  is  a  valid  stimulus.  Distributed  architectures  can  provide  robust  test  environments  and  offer 
opportunities  to  conduct  concurrent,  rather  than  sequential  test  events,  that  generate  actual  SUT 
performance  data.  Distributed  testing,  as  we  see  it,  is  a  subset  of  ADS.  There  will  almost 
certainly  be  opportunities  to  use  ADS  as  a  simulation  tool  in  support  of  acquisition  programs,  but 
there  will  also  be  potential  benefits  to  using  distributed  architectures  as  genuine  test  tools.  In  this 
section  the  lessons  learned  from  all  three  JADS  tests  are  compiled  under  technical  and 
programmatic  categories.  In  some  cases  the  lessons  learned  apply  to  more  than  one  category  and 
thus  appear  multiple  times.  For  each  category  an  attempt  was  make  to  distill  the  lessons  learned 
in  that  category  into  some  key  findings.  The  process  to  do  this  was  very  subjective  and  the 
reader  is  encouraged  to  perform  his/her  own  assessment. 

3.1  Technical 

3.1.1  Simulations 

-  Careful  design  and  thorough  integration  testing  are  required  regardless  of  whether  you’re 
developing  and  linking  a  new  simulation  or  linking  an  existing  stand-alone  simulation. 

-  As  in  the  SIT  LSP,  a  major  lesson  learned  is  that  stand-alone  simulation  facilities  (for  live, 
virtual  or  constmctive  entities)  can  require  significant  modifications  before  effective  linking 
is  possible. 

-  Simulations,  when  used  in  distributed  testing,  should  be  carefully  planned  and  developed. 
Using  reliable  simulations  is  important  for  a  successful  test. 

-  Some  problems  existed  in  the  robustness  of  the  Digital  System  Model  (DSM)  software  that 
resulted  in  reliability  problems  with  the  DSM  and  its  federate.  Although  a  hindrance,  the 
DSM  performance  was  adequate  to  collect  the  needed  data.  Better  software  design,  quality 
assurance,  and  testing  would  have  uncovered  the  reliability,  logic,  and  buffer  limit  issues 
seen  in  the  test  execution. 

3.1.2  Interfaces 

-  For  the  foreseeable  future,  distributed  testing  will  utilize  simulations  that  were  not  designed 
to  work  together.  In  most  cases  it  is  more  cost  effective  to  develop  interface  units  than  it  is  to 
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modify  existing  simulations.  Careful  design  and  testing  of  these  interfaces  are  required  to 
ensure  they  perform  the  desired  functions  without  adding  large  processing  delays. 

—  Accurate  coordinate  transformations  are  necessary.  They  must  be  verified  and  validated  at 
each  site  and  then  revalidated  during  end-to-end  testing  as  early  as  possible  in  the  test  phase. 

—  Network  interface  units  (NIU)  need  improvement.  NIUs  are  necessary  if  two  nodes  cannot 
communicate  directly  in  a  common  language.  They  can  be  a  major  source  of  both  errors  and 
processing  delays.  Better  direct  user  control  of  the  content  of  the  data  and  network 
communications  is  needed. 

—  Linking  of  facilities  using  ADS  can  require  significant  facility  interface  hardware  and 
software  development.  ADS  implementation  is  not  “plug  and  play,  at  least  for  some  time. 

“  Additionally,  linking  may  require  special  purpose  interfaces  to  accept  inputs  in  real  time. 
Development  of  such  units  must  be  factored  into  test  planning. 

—  Latency  variations  were  significant.  Processing  delays  were  the  primary  culprit  here. 

—  Key  interfaces  need  realistic  integration  testing.  Replaying  data  from  a  recorded  mission 
worked  well  in  most  cases  (and  was  most  cost  effective);  however,  some  integration  testing 
required  a  live  mission. 

—  Distributed  testing  often  requires  linkage  among  dissimilar  facilities,  network  equipment,  and 
simulations.  However,  careful  planning  can  significantly  reduce  the  potential  for  difficulties 
arising  from  network  interface  problems. 

3.1.3  Networks 

—  Distributed  testing  applications  have  unique  and  sometimes  stringent  network  performance 
requirements  for  latency  and  latency  variation.  There  are  many  design  decisions  that  impact 
network  performance.  Special  instrumentation  and  thorough  testing  are  required  to  ensure 
the  network  is  meeting  performance  requirements. 

—  Common  ADS-related  hardware  and  software  are  needed.  In  the  LSP,  it  was  difficult  to  get 
the  ADS  network  to  behave  in  a  uniform  fashion  because  of  the  many  different  types  of 
interface  hardware,  communications  equipment  (routers),  and  interface  software  versions. 
Latency  variations  were  significant.  Processing  delays  were  the  primary  culprit  here. 

—  Early  definition  of  network  requirements  was  very  advantageous.  This  was  a  major  lesson 
from  the  LSP  that  JADS  took  advantage  of. 

—  Federations  should  experiment  with  different  transport  modes  to  determine  the  optimum  mix 
of  transport  modes.  In  the  EW  test  federation  performance  varied  as  the  mix  of  reliable  and 
best  effort  data  changed.  Through  trial  and  error,  fewer  problems  with  latency  and  data  loss 
were  noted  if  less  reliable  traffic  was  published  within  the  federation.  However,  this  was  a 
subjective  opinion  because  no  tools  were  available  to  test  the  performance  envelope  of  the 
architecture.  Federation  performance  was  poor  when  integration  testing  started.  RTI 
developers  should  have  tools  or  performance  measurements  to  guide  federation  developers  as 
they  design  and  integrate  their  architectures.  The  link  health  messages  were  required  to  be 
published  as  best  effort  to  correct  this  problem.  This  change  was  made  late  in  the  integration 
effort  to  further  tune  the  architecture  with  the  real  federates.  Link  health  check  messages 
were  published  best  effort  during  both  ADS  testing  phases. 
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3.1.4  Instrumentation 


-  Special  network  instrumentation  is  required  to  monitor  network  performance  during  both 

development  and  test  execution.  Time  stamping  and  synchronization  are  unique  aspects  of 
distributed  testing  that  required  special  attention.  Live  players  also  have  unique 

instrumentation  requirements  for  distributed  test  applications. 

-  Time  sources  must  be  synchronized  off  the  same  time  source  and  then  must  be  validated  at 

each  test  site  prior  to  project  operations  to  ensure  accurate,  synchronized  time  is  precisely 

recorded  at  each  test  site. 

-  Special  test  equipment  is  needed  for  checkout  and  verification  of  the  ADS  architecture. 
Without  this  equipment,  trial  and  error  becomes  the  norm  when  (not  if)  problems  crop  up. 

-  Changes  and  upgrades  to  aircraft  instrumentation  delayed  development.  Specially 

instrumented  aircraft  were  required  to  support  the  LFP  flights.  Because  of  the  small  number 
of  such  aircraft,  the  LFP  schedule  was  very  sensitive  to  periodic  aircraft  phase  inspections, 
software  upgrades,  and  higher  priority  missions. 

-  In  the  LSP  merging  several  TSPI  sources  was  advantageous.  Real-time  aircraft  inertial 
navigation  system  (INS)  and  GPS  data  were  combined  to  calculate  more  accurate  kinematic 
estimates.  When  combined  with  the  ground  radars,  solutions  of  one  to  three  meters  in 
position  and  one  meter  per  second  in  velocity  were  achieved. 

-  Time  sources  must  be  synchronized  using  a  “master  clock”  and  then  validated  at  each 
network  node. 

-  Special  test  equipment  and  networking  tools  are  necessary  for  distributed  testing.  The  tool 
set  must  be  able  to  rapidly  isolate  the  specific  cause  of  network  and  ADS/DIS  problems. 

-  In  the  EW  test  TSPI  data  losses  among  platform  federate,  the  DSM,  AFEWES,  and  test 
control  federate  (TCF)  occurred  frequently  during  federation  integration.  JADS  and  DMSO 
investigated  this  problem.  A  work-around  solution  was  a  fixed  join  process  for  federates 
prior  to  each  test  run.  After  the  solution  was  implemented,  data  losses  among  the  DSM, 
AFEWES,  and  the  platform  federate  were  still  observed.  These  losses  manifested  themselves 
in  the  apparent  hovering  aircraft  observed  in  the  federate  integration  test  (FIT).  It  was  not 
clear  what  caused  the  data  loss,  but  dead  reckoning  aircraft  position  provided  an  acceptable 
solution.  This  data  loss  coupled  with  the  dead  reckoning  implementation  at  AFEWES  was 
the  suspected  cause  of  an  extremely  large  miss  distance  value  on  one  missile  shot.  RTI 
bundling  of  federate  data  for  transmission  made  troubleshooting  data  flow  and  transmission 
problems  more  difficult.  Our  tools  assessed  hardware  performance  only. 

-  Instrumentation  for  federation  performance  evaluation  used  in  the  EW  Test  Phase  2  was 
inadequate.  It  lacked  the  ability  to  examine  data  passed  between  RTI  instances.  Best  effort 
data  could  be  dropped  by  the  network  without  notification  or  without  any  faults  reported  by 
the  hardware.  Phase  3  network  instrumentation  must  be  expanded  to  include  network 
sniffers  to  monitor  network  traffic  between  the  sites. 

-  In  the  EW  test  JADS  was  highly  dependent  upon  time  synchronization  of  all  federate 
computers  and  software.  Any  requirement  for  synchronization  requires  the  ability  to  verify 
that  the  requirement  is  being  met.  For  example,  if  one  millisecond  synchronization  accuracy 
is  required,  then  a  capability  to  measure  time  between  two  computers  at  one  millisecond 
precision  is  necessary.  However,  software  tools  were  not  available  to  measure  accuracy  at 
that  level.  In  fact,  testing  time  synchronization  across  the  federation  was  more  art  than 
science.  Even  with  time  synchronization  and  the  time  cards  implemented  in  all  computers. 
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instances  were  noted  where  time  synchronization  “slipped”  affecting  latency  measurements. 
A  few  occurrences  of  time  synchronization  problems  across  the  federation  were  observed 
requiring  JADS  to  research  particular  runs  after  daily  testing.  JADS  consistently  followed 
documented  time  synchronization  procedures  and  hardware  settings.  Site  support  personnel 
were  relied  upon  to  implement  procedures  and  verify  settings  daily. 

If  you  are  going  to  use  hardware  for  time  synchronization  (e.g.,  BanComm  cards)  obtain  time 
directly  from  the  card.  You  may  have  to  write  device  drivers  to  get  this  capability.  You  also 
need  to  resolve  how  you  will  measure  time  synchronization  differences.  However,  other 
alternatives  exist.  The  network  time  protocol  (NTP)  software  (xntp  for  UNIX  hosts,  NTP 
time  for  PC  hosts)  provided  an  easy-to-use  method  to  synchronize  system  clocks  to  a  time 
source.  In  other  JADS  tests  the  system  clock  could  be  kept  within  one  millisecond  of  the 
time  source.  The  software  does  require  time  and  attention  to  reach  this  level  of  performance. 

However,  it  keeps  statistics  on  how  well  it  is  keeping  time  and  it’s  free. 

In  the  EW  test  ADS  was  not  able  to  completely  solve  time  synchronization  issues  in  the 
federate  computers  using  time  cards.  In  theory,  the  hardware  cards  should  provide  the  most 
accurate  time  synchronization  available.  In  practice,  some  implementations  proved  more 
robust  than  others  did  and  verifying  time  synchronization  across  a  wide  area  network  (WAN) 
proved  to  be  elusive.  The  most  effective  configuration  of  the  BanComm  cards  was  not 
implemented  for  time  synchronization  on  either  the  UNIX-based  or  the  personal  computer 
(PC)-based  hosts.  In  addition,  there  were  problems  with  the  BanComm  hardware,  BanComm 
software,  and  with  one  of  JADS  contractor’s  attempts  to  write  software  to  use  the  BanComm 
cards.  The  software  executing  on  the  Silicon  Graphics.  Inc.,  (SGI)  02s  read  time  directly  off 
the  BanComm  cards  via  the  JADS  contractor-developed  driver  software.  This  provided  the 
most  accurate  time  synchronization  solution.  However,  the  method  used  to  obtain  tinie 
information  (via  overloading  of  an  IRIX  operating  system  call)  had  the  limitation  that  it  did 
not  provide  any  means  for  the  federates  to  query  the  BanComm  card  as  to  whether  it  was 
actually  using  the  Inter-Range  Instrumentation  Group  (IRIG)-  B  time  code  input  signal  (the 
desired  state)  or  free  running  using  its  internal  crystal  oscillator.  However,  on  the  PCs, 
BanComm-provided  software  was  used  to  synchronize  PC  system  time  to  BanComm  card 
time.  This  was  not  very  accurate,  and  in  some  cases,  time  on  the  PCs  was  off  by  as  much  as 
60  milliseconds.  In  addition,  this  software  did  not  synchronize  the  system  time  immediately 
when  Windows  95  or  Windows  98  was  started  or  restarted,  which  apparently  caused  several 
aborted  runs  because  of  the  time  on  an  ADRS  PC  being  unsynchronized.  Also,  for  the  PCs, 
there  was  still  the  problem  of  determining  when  the  BanComm  lost  its  signal  and  was  free 
running  on  its  internal  oscillator.  Finally,  JADS  lacked  an  adequate  method  of  detecting  time 
synchronization  problems  in  real  time  during  federate  execution  runs.  Only  in  cases  where 
severe  symptoms  were  produced  by  time  synchronization,  problems  such  as  bursts  of 
platform  federate  live  entity  state  and  threat  performance  messages  caused  by  a  start  time  m 
the  past  (for  platform)  were  noticed  immediately  and  corrected.  Data  that  were  time  stamped 
on  the  PCs  (ADRS  and  DSM)  were  only  judged  “good  enough.”  There  was  a  lot  of  variation 
in  the  time  value  that  originated  on  the  PCs.  This  did  not  impact  the  ADRS  PCs  as  they  only 
used  the  time  stamp  in  the  start  command,  telling  all  federates  to  start  at  some  time  in  the 
future.  However,  the  DSM  PC  did  exhibit  some  odd  behavior  that  affected  calculation  of 
jammer  response  times. 
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3.1.5  Test  Control 


-  There  are  many  unique  aspects  to  distributed  test  control  that  require  a  carefully  selected  mix 
of  central  and  local  control.  The  central  control  facility  must  have  the  display  and 
communications  capabilities  to  know  total  system  health  in  real  time.  Total  system  health 
includes  not  only  the  status  of  the  real,  virtual,  and  constmctive  players  but  also  the  data 
processing  and  collection  systems  and  the  system  synchronization  mechanism.  Real-time 
processing  of  system  data  is  essential  to  efficient-test  conduct. 

-  Test  communications  requirements  must  be  addressed  early  in  the  test  planning  phase.  This 
is  necessary  to  ensure  effective  communications  during  the  test.  Also,  a  linked  test  should 
have  multiple  (more  than  two)  communications  nets  with  easy,  selectable  access  to  all  the 
nets  from  multiple  locations  within  the  site.  Finally,  the  capability  for  secure  video 
teleconferencing  pays  big  dividends  during  planning,  coordination,  and  post-test  debriefs. 

-  Local  (on-site)  test  monitoring/control  should  be  used  prior  to  remote  test  monitoring/control. 

-  Tight  control  of  the  aircrew  is  not  desirable.  Give  them  the  critical  parameters  and 
switchology  to  meet  the  test  objectives  and  allow  them  to  make  tactical  decisions,  fly  the 
“aircraft,”  operate  the  weapon  system,  etc. 

-  Several  subnetworks  should  be  used  for  voice  communications.  Three  voice 
communications  networks  were  needed  to  support  more  than  30  people  at  various  locations, 
and  a  fourth  network  would  have  further  aided  decision  making. 

-  Two-dimensional  displays  were  needed  at  each  node;  they  greatly  enhanced  the  situational 
awareness  of  the  participants. 

-  Have  a  centralized  test  control  center  with  test  controllers  who  are  extremely  familiar  with 
the  test  and  network  configuration. 

-  Personnel  involved  in  a  distributed  test  need  to  understand  the  “big  picture.”  When  people 
are  geographically  separated,  it  becomes  easy  for  them  to  focus  on  their  own  individual 
portion  of  the  test.  When  problems  arise,  personnel  who  understand  the  entire  test  and  the 
overall  network  will  find  solutions  much  faster. 

-  Operator  boredom  with  the  repetitiveness  of  test  runs  at  AFEWES  may  have  contributed  to 
afternoon  run  differences.  JADS  attempted  to  minimize  tum-around  time  between  runs.  The 
time  between  mns  from  start  time  to  start  time  was  usually  between  6  and  8  minutes. 

-  Detailed  planning  for  run  management  is  necessary  before  testing  commences.  During  daily 
testing,  a  run  matrix  was  used  to  determine  which  runs  were  executed;  what  procedures  were 
used  to  start  the  federates,  models  and  simulators;  and  to  initiate  the  run,  stop  the  run  and 
shut  down  of  the  federates.  The  work  done  during  preliminary  testing  (e.g.,  federate 
acceptance  test  [FAT],  federate  integration  test  [FIT])  provided  a  repeatable  methodology  for 
orderly  federation  operation  that  was  used  for  the  formal  test.  Nonetheless,  problems  were 
frequently  encountered,  errors  were  made,  and  unanticipated  issues  arose. 

-  During  test  runs,  the  TCAC  test  controller  was  highly  dependent  on  ADRS  for  federation  and 
scenario  status  monitoring.  Analysts  were  highly  dependent  on  the  ADRS  emitter  state 
history  display  for  monitoring  jammer/threat  engagement  details.  JADS  found  the  analysis 
federate  scenario  visualizer  to  be  a  solid  capability.  While  it  provided  an  extra  set  of 
graphical  displays  of  the  unfolding  engagement,  it  also  provided  real-time  feedback  of  the 
EW  Test  MOPs.  Anomalies  in  miss  distance  and  response  times  could  be  instantly  assessed, 
which  was  an  added  capability  separate  from  ADRS.  The  analysis  federate  could  not  take 
the  place  of  an  ADRS  machine  for  Phase  3,  but  it  could  support  troubleshooting  of  anomalies 
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seen  during  the  runs.  Without  the  analysis  federate,  problems  seen  could  be  at  AFEWES, 
ACETEF  or  in  one  of  the  many  federates  run  at  JADS.  The  analysis  federate  aided  in 
identifying  the  source  of  the  problem.  If  presented  with  extremely  limited  time  and  manning, 
the  analysis  federate  could  be  eliminated  with  only  a  small  impact  to  test  execution.  It  was 
not  critical  to  the  function  of  the  test  but  did  provide  an  extra  source  of  examination  of  the 
run  execution.  The  greatest  benefit  of  the  analysis  federate  was  the  real-time  assessment  of 
the  EW  Test  MOPs  and  the  integration  of  the  aircraft  profile  with  the  threat  mode  status.  If 
tasks  needed  to  be  combined,  the  analysis  federate  would  need  to  be  updated  to  assume  the 
responsibilities  of  the  second  ADRS  machine. 

JADS  EW  voice  links  were  conducted  using  conference  calls  with  open  lines  to  AFEWES 
and  ACETEF.  This  capability  continued  to  evolve  as  command  and  control  requirements 
evolved.  JADS  used  head-mounted  earphone/microphone  equipment  and  experienced 
numerous  problems  with  hearing  and  being  heard  across  the  network.  Most  problems  were 
alleviated  with  equipment  familiarity  and  experience.  Batteries  had  to  be  replaced 
frequently.  The  FAT  and  FIT  demonstrated  shortfalls  in  the  voice  communications  that 
required  more  equipment  at  each  facility.  As  the  number  of  instruments  increased,  testers 
became  very  busy  coordinating  status,  communicating  test  information,  and  eontrolling  run 
execution.  Consequently,  message  transmission  length  had  to  be  minimized;  external 
background  conversations  avoided;  and  test  problem  troubleshooting  had  to  be  done  via  a 
separate  line.  ACETEF  had  some  telephone  instmment  problems.  JADS  used  Phase  2 
conference  call  initialization  methods  during  Phase  3  and  included  more  formalized 
discussion  procedures  and  protocols. 

JADS  refined  voice  protocols  for  acknowledging  readiness  among  sites  and  starting/stopping 
runs  for  EW  Phase  3.  Further  review  of  link  health  status  confirmation  procedures  and 
network  health  check  impacts  was  needed.  JADS  improved  situational  awareness  for 
network  health  and  readiness  across  sites  and  formalized  the  procedures  as  necessary. 

The  health  check  was  inadequate  for  monitoring  JADS  EW  federation  status  for  several 
reasons.  First,  its  time  scale,  which  was  about  20  seconds  before  any  indication  of  a 
problem,  was  too  long  for  a  real-time  federation.  Second,  because  it  sent  its  messages  via 
TCP/Intemet  protocol  (IP),  this  system  could  not  detect  a  problem  for  federate  messages  sent 
via  RTI  best  effort,  i.e.,  UDP/IP-based  protocol,  unless  the  underlying  cause  of  the  problem 
affected  both  of  those  protocols.  And  third,  the  RTI  did  not  time  stamp  and  log  these 
messages,  so  they  were  only  available  in  real  time.  JADS  did  not  use  this  as  a  primary 
indication  of  federation  health,  so  no  changes  were  required.  Future  federations  should 
investigate  RTI  tuning  features  (e.g.,  runtime  infrastructure  initialization  data  [RID]  file 
parameters)  or  other  RTI  management  features  (e.g.,  management  object  model  calls)  if  the 
federation  doesn’t  implement  its  own  health  monitors  like  JADS  did. 

The  JADS  link  health  check  (LHC)  scheme  during  the  EW  test  provided  reasonable  insight 
into  federation  health  once  it  was  understood.  Analysis  showed  that  there  was  a  high 
correlation  between  the  loss  of  LHC  messages  and  most,  but  not  all,  events  that  involved  the 
loss  of  other  federate  messages  sent  best  effort  and/or  the  delay  of  messages  sent  via  RTI 
reliable,  TCP/IP-based  communications  protocol.  Due  to  its  1  hertz  message  frequency,  the 
LHC  system  sometimes  missed  best  effort  data  loss  events  lasting  less  than  1  second,  but 
those  events  apparently  did  not  cause  any  simulation  problems.  Since  the  LHC  system  sent 
its  messages  via  the  best  effort  protocol,  it  could  also  not  detect  short-duration  problems  that 
affected  only  the  TCP/IP  connection  used  for  reliable  protocol  between  two  federates. 
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Perhaps,  the  most  interesting  result  from  the  post-test  analysis  of  the  LHC  messages  was  that 
the  LHC  system  detected  selective,  one-way  best  effort  data  losses  between  federates  that 
may  be  a  symptom  of  problem(s)  with  the  RTIs  use  of  IP  multicast  groups.  For  the  runs 
■during  which  these  problems  were  observed,  the  losses  were  selective  because  LHC 
messages  (and  usually  other  federate  messages  sent  best  effort  as  well)  were  lost  between  one 
or  more  federates  at  JADS  and  the  federate  at  another  test  node  but  not  between  the 
remaining  JADS  federates  and  that  remote  federate.  The  losses  were  one  way  because  the 
LHC  messages  between  the  federates  experiencing  the  problem  were  lost  in  only  one 
direction.  Typically,  during  such  events,  there  was  no  delay  in  the  flow  of  reliable  messages 
between  those  federates,  if  any  reliable  traffic  was  present.  It  was  difficult  to  understand 
how  network  or  network  hardware  problems  could  produce  such  selective,  one-way  data 
losses.  While  LHC  as  implemented  has  limitations,  it  was  sufficient  for  JADS  in  Phase  3. 
Future  federations  should  consider  the  limitations  noted  if  they  choose  to  pursue  a  similar 
health  monitor  scheme  for  their  federation. 

Test  design  should  include  as  much  real-time  analysis  as  possible.  This  will  increase  the 
success  rate  during  test  execution  and  minimize  time  required  to  repeat  test  activities  if  the 
equipment  problems  are  not  noticed  until  after  test  completion.  Future  testers  should  also 
consider  possible  actions  when  problems  are  discovered  during  test  execution.  The  decision 
to  stop  testing  until  the  problem  is  fixed  or  continue  and  account  for  the  problem  in  the 
results  is  a  difficult  decision  that  should  be  considered  long  before  test  execution  begins. 
During  the  various  test  phases,  real-time  analysis  became  more  and  more  crucial  to  the 
successful  execution  of  the  test.  EW  Phase  3  was  largely  impacted  by  the  need  to 
accomplish  as  many  successful  runs  as  possible  in  the  least  amount  of  test  time.  Without  the 
ability  to  observe  and  critique  performance  from  the  various  federation  participants,  the  test 
time  would  have  been  lengthened  or  the  useable  test  data  collected  would  have  been 
significantly  decreased.  ADRS  and  site  observers  were  vital  to  correcting  operator  actions 
and  clarifying  the  rules  of  engagement.  SUT  observers  were  also  critical  in  determining  if 
the  SUT  was  performing  as  desired  as  the  test  was  executed.  Network  observers  were  also 
needed  to  observe  the  performance  of  network  equipment  during  test  execution.  The  ability 
in  all  phases  to  watch  threat  performance,  operator  performance,  and  SUT  performance 
became  a  cornerstone  to  successful  test  execution.  The  real-time  analysis  supplied  vital 
information  to  the  test  controller  who  could  ask  questions  about  specific  equipment  or 
operators  as  soon  as  problems  were  noticed.  During  Phase  3,  this  capability  corrected  severe 
operator  training  problems  at  AFEWES  and  SUT  problems  at  ACETEF  that  allowed  many 
runs  to  be  saved  in  the  final  data  sets. 

ADRS  equipment  crashes  and  reboots  frequently  disrupted  testing  and  slowed  the  rate  of 
JADS  EW  testing.  The  problem  was  moderated  during  Phase  2  by  adopting  new  procedures 
like  rebooting  each  time  a  computer  was  idle.  SGI  02  to  PC  interface  software  developed 
for  our  federates  (called  the  JADS  communicator)  would  leave  a  communications  socket 
allocated  after  ADRS  crashed,  so  additional  time  was  lost  waiting  for  the  socket  to  reset. 
Procedural  speed  for  starting  ADRS  contributed  to  the  problem.  It  was  very  important  that 
the  computer  be  started  in  a  specific  sequence  in  the  TCAC.  The  action  adopted  by  JADS 
was  to  analyze  the  run  logs  for  reliability  problems  to  determine  where  changes  in  processes 
and  communication  could  improve  Phase  3  operations.  A  memory  leak  problem  was 
detected  in  the  ADRS  computer  that  resulted  in  many  software  crashes  caused  when  the 
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system  would  run  out  of  available  memory.  This  was  solved  by  the  frequent  reboots.  It  was 
also  marked  for  correction  before  Phase  3. 

3.1.6  Software  Development 

-  Testing  in  software  quality  is  not  generally  possible.  Poorly  designed  software  r^ely 
emerges  from  testing  in  any  better  condition.  Conversely  one  should  not  take  the  position 
that  well  planned  and  developed  software  does  jiot  require  testing.  The  development  and 
coordination  of  complex  models  to  support  distributed  testing  requires  extraordinary 
attention  to  configuration  management  issues.  The  added  complexity  of  distributed  testing 
over  stand-alone  applications  makes  following  proven  software  engineering  standards  and 
procedures  critical.  Configuration  control  was  particularly  problematic  and  required 
extraordinary  attention. 

-  Some  problems  existed  in  the  robustness  of  the  EW  DSM  software  that  resulted  in  reliability 
problems  with  the  DSM  and  its  federate.  Although  a  hindrance,  the  DSM  performance  was 
adequate  to  collect  the  needed  data.  Better  software  design,  quality  assurance,  and  testing 
would  have  uncovered  the  reliability,  logic,  and  buffer  limit  issues  seen  in  the  test  execution. 

-  Configuration  control  is  essential.  This  one  obvious  area  was  one  of  great  challenp  for  all 
three  JADS  tests  considering  the  many  sites  involved  and  the  multiple  uses  of  each  site. 

-  Configuration  changes  on  tools  (analysis  federate,  ADRS  display,  DSM  joining  process, 
AFEWES  dead  reckoning  algorithms)  could  have  severe  impacts.  Configuration  changes, 
even  seemingly  trivial  ones,  must  be  coordinated  at  all  levels.  JADS  stressed  configuration 
management  procedures  for  Phase  3  and  enforced  their  use. 

-  Acceptance  testing  of  federate  software  is  a  recommended  practice.  These  acceptance  tests 
should  be  designed  to  (1)  test  the  software  in  its  intended  mode  of  operation,  and  (2)  test  all 
requirements  of  the  software.  It  can  encourage  the  developer  to  fix  problems  before  they 
impact  the  test.  It  provides  an  excellent  mechanism  for  supporting  the  V&V  of  the  federation 
by  proving  the  federates  are  built  correctly  and  satisfy  the  needed  simulation  requirements. 
Additionally,  acceptance  tests  provide  a  clear  event  to  which  configuration  management 
milestones  can  be  tied.  It  was  JADS  experience  that  software  acceptance  testing  of  ADS 
components  in  their  stand-alone  mode  did  not  uncover  problems  encountered  once  they  were 
integrated  into  the  ADS  environment.  Originally  software  acceptance  testing  was  not 
planned  as  part  of  JADS  software  development  effort  for  the  EW  test.  Formal  testing  was 
thought  to  be  too  costly  and  too  late  in  the  development  process  to  be  effective.  JADS 
planned  to  use  in-process  reviews  with  each  developer  to  gain  insight  and  cross 
communication  to  get  the  right  software  products  developed.  However,  when  JADS  was 
unable  to  gain  insight  into  the  software  development  and  received  obvious  indications  that 
there  were  flaws  in  some  of  the  software  items,  JADS  elected  to  use  acceptance  testing. 
Because  of  cost  and  schedule  constraints,  the  scope  of  these  tests  was  limited  to  the 
development  environments  and  to  the  test  sets  that  were  available  at  the  time.  These 
acceptance  tests  did  not  address  all  software  requirements.  For  example,  the  acceptance  test 
did  not  consider  the  operational  modes  of  the  jammer  DSM  as  executed  in  the  ADS 
environment.  The  acceptance  testing  also  did  not  stress  the  model  to  the  level  of  execution 
encountered  within  the  ADS  test  environment.  This  resulted  in  a  model  that  functioned  well 
in  stand-alone  mode  but  was  marginal  when  integrated  into  the  ADS  environment  and 
operated  according  to  the  test  procedures.  Acceptance  testing  provided  a  more  solid  basis  for 
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V&V  efforts.  The  limited  acceptance  test  addressed  several  key  requirements  such  as  correct 
calculation  of  received  power  and  correct  calibration.  The  results  of  the  acceptance  test  were 
available  to  the  accreditation  board  to  determine  if  the  software  met  JADS’  needs.  Finally, 
acceptance  testing  allowed  a  convenient  point  to  establish  configuration  baselines  and  to 
transfer  control  of  those  baselines  to  JADS.  Acceptance  testing  was  better  planned  in  Phase 
3  of  the  EW  test  even  though  we  still  had  limited  test  cases  and  tools.  The  new  software  was 
acceptance  tested  as  part  of  the  V&V  plan.  Formal  baselines  were  established  after 
completion  of  the  acceptance  tests. 

Frequently,  the  government  only  knows  in  general  terms  what  is  needed  to  execute  tests  in  a 
geographically  distributed  environment.  Test  design  must  mature  to  identify  the  specific 
capabilities  that  each  facility  will  provide  before  specifications  are  created.  This  generally 
precludes  creating  good  performance  specifications  prior  to  contract  award.  Sufficient 
tasking  must  be  included  in  the  SOW  to  ensure  that  government  interests  are  covered  and  the 
lead  from  the  contractor  has  a  leverage  tool  to  use  to  ensure  the  work  is  executed  on  time 
with  good  quality.  Sequential  contract  awards  may  be  used  to  mitigate  risks  associated  with 
loose  SOWs.  JADS  learned  from  the  planning  and  execution  of  the  EW  test  that  abbreviated 
statements  of  work  (SOW)  and  reduced  deliverables  resulted  in  differences  in  expectations 
between  contractors  and  the  government.  The  loosely  defined  SOW  allowed  the  analysis 
team  to  continue  to  refine  software  requirements  for  a  critical  piece  of  software  necessary  for 
the  test  execution  well  beyond  the  date  it  should  have  been  finalized.  Several  measures  of 
performance  were  modified  from  the  non-traditional  calculation  to  help  measure  ADS 
effects.  This  proved  more  difficult  than  expected.  Since  delivery  schedules  were  not  clearly 
defined,  the  contractor  permitted  these  discussions  to  go  on  well  beyond  the  time  needed  to 
code  and  test  the  software  to  meet  the  government’s  expected  delivery  date.  All  parties  were 
trying  to  get  the  best  insight  into  ADS  effects  on  the  EW  Test  measures  while  balancing 
impacts  to  the  software.  The  problem  was  resolved  when  the  government  program  manager 
froze  software  requirements  and  provided  the  contractor  a  specific  delivery  date.  A  second 
impact  was  related  to  the  level  of  on-site  test  support.  The  loosely  defined  SOW  allowed  the 
contractor  to  reallocate  on-site  resources  earlier  in  the  test  design  to  support  other  test 
activities.  The  reallocation  was  discussed  with  the  government;  however,  the  impact  to  on¬ 
site  support  during  Phase  2  was  not  explicitly  negotiated.  As  a  result,  the  government 
received  less  support  than  expected.  Once  the  software  requirements  were  frozen,  the 
updated  software  was  delivered  in  time  for  test  execution. 

Strict  contractual  requirements  may  be  needed  for  organizations  where  the  development 
processes  are  not  well  understood.  Critical  software  should  be  developed  by  companies  with 
proven  subject  matter  experience  and  sound  software  development  practices.  Well-defined 
quality  software  practices  were  important  for  any  software  development;  however,  when 
working  with  multiple  facilities  in  an  ADS  test,  strict  adherence  to  practices  were  necessary 
to  ensure  success.  In  addition,  processes  for  assessing  software  quality  (e.g.,  independent 
acceptance  test)  were  needed  to  ensure  that  each  ADS  component  operated  as  expected.  No 
plan  existed  to  ensure  software  quality  for  the  JADS  EW  test.  JADS  originally  relied  on 
each  developer’s  internal  practices  to  produce  quality  software.  JADS  attempted  to  gain 
insight  into  software  development  at  each  facility  but  failed.  Post-development  quality 
measures  were  implemented  to  inspect  delivered  software.  Several  problems  were  identified 
with  the  DSM  that  should  have  been  identified  earlier  in  the  software  development  process. 
Specifically,  software  requirement  specifications  (SRS)  and  the  interface  control  document 
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(ICD)  were  sometimes  misinterpreted  by  the  developers.  These  problems  could  have  been 
found  by  closer  monitoring  of  the  software  development  process,  particularly  in  the  area  of 
requirements  management.  As  we  learned  these  valuable  lessons  JADS  became  more 
involved  in  the  software  development  process  of  the  remaining  federates.  Daily  contact 
prevented  several  errors  from  going  undetected  and  resolved  the  problems  before  the  actual 
test  event. 

Perhaps  the  most  important  lesson  learned  from  EW  Phase  2  and  the  preparation  for  it  was 
the  critical  importance  of  careful  planning  and  preparations  at  the  earliest  stages  of  the 
program.  It  is  better  to  avoid  problems,  since  there  may  not  be  enough  time  and/or  money  to 
find  and  fix  them  later.  This  seems  especially  true  for  ADS  programs.  The  nature  of  ADS 
brings  multiple  facilities  together,  each  having  their  own  development  style  and  practices  and 
each  bringing  a  potentially  different  understanding  of  the  problem.  This  is  very  similar  to 
having  multiple  facilities  working  together  to  develop  a  single  software  package.  Any 
actions  that  reduce  ambiguity  in  the  interface  design  will  reduce  the  risk  of  the  program. 
This  is  very  important  for  ADS-based  tests  since  it  may  be  difficult  to  slip  test  schedules 
when  multiple  facilities  are  involved.  Hence,  the  importance  of  a  good  ICD  and  enforcing 
the  same  methods  of  compliance  from  the  start  of  software  development.  An  ICD  was 
developed  for  the  JADS  federation  to  guide  software  developers.  Two  problems  were 
identified  relating  to  this  ICD:  (1)  nonconformance  to  the  ICD  and  (2)  differences  in 
interpretation  of  complex  concepts.  Prior  to  the  test,  the  description  of  the  coordinate 
transformation  was  agreed  to  be  acceptable  by  all  participants;  however,  facilities  developed 
different  implementations  of  the  software  when  coding  was  finished.  The  problem  was 
finally  resolved  when  JADS  provided  sample  transformation  pairs  for  testing  each  facility’s 
algorithm.  These  sample  data  points  should  have  been  included  in  the  JADS  ICD  to  avoid 
confusion.  In  some  instances  software  was  developed  that  did  not  conform  to  the  ICD. 
Because  of  the  lack  of  detailed  acceptance  testing  these  nonconformance  problems  were  not 
found  until  very  late  in  the  integration  process.  As  a  result,  decisions  had  to  be  made  either 
to  bring  the  software  into  conformance  or  to  change  the  ICD  in  order  to  maintain  test 
schedule.  For  example,  problems  with  the  federate  message  sequence  numbers  illustrate  both 
the  test  and  post-test  impacts.  For  each  instance  of  a  simulation  object,  the  federates  should 
have  used  sequence  numbers  in  outgoing  messages  stcirting  from  1  and  incremented  by  1  for 
each  successive  message.  However,  because  of  a  combination  of  ambiguous  ICD  wording 
and  lack  of  early  ICD  compliance  testing  and  enforcement,  the  TTH  and  AFEWES  federates 
transmitted  message  sequences  that  did  not  conform  to  the  same  sequence  numbering 
scheme.  During  Phase  2  test  execution  this  became  a  problem  with  the  DSM  PC’s  real-time 
error  checking  for  incoming  source  mode  change  (SMC)  messages.  The  sequence  number 
was  used  by  the  DSM  to  detect  missing  and  out-of-order  messages.  Since  the  sequence 
numbers  were  not  set  correctly,  the  error  reports  were  misleading  and  ineffective.  During  the 
post-test  analysis,  improper  message  sequence  numbers  for  several  message  types  made  it 
more  difficult  to  detect  and  analyze  runs  with  data  loss  and  latency  problems  for  the  ADS 
analysis  process.  In  particular,  it  greatly  complicated  the  calculation  of  overall  latencies  for 
the  critical  combination  of  outgoing  SMC  messages  and  the  corresponding  jammer  technique 
command  messages  generated  by  the  DSM.  To  correct  for  these  problems  message  sequence 
counters  were  corrected  for  both  the  TTH  and  AFEWES  threat  federates  for  Phase  3. 
Wording  in  the  ICD  was  changed  to  be  less  ambiguous. 
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3.1.7  Integration 

-  Time  sources  must  be  synchronized  using  a  “master  clock”  and  then  validated  at  each 
.network  node. 

-  Get  SUT  experts  involved  from  the  beginning. 

-  A  stepped  build-up  approach  should  be  used.  First,  a  systematic  checkout  of  the  stand-alone 
simulators  (live,  virtual  or  constructive)  is  needed.  Next,  direct  (non-DIS)  links  should  be 
used  during  test  build-up.  Finally,  structured  testing  of  the  network  must  be  performed  prior 
to,  and  independent  of,  the  linked  testing  times  and  the  simulation  laboratories  to  validate 
transmission/reception  rates,  bandwidth  utilization,  latency,  data  transmission  and  reception, 
etc.,  prior  to  commencing  project  test  periods. 

-  Key  interfaces  need  realistic  integration  testing.  Replaying  data  from  a  recorded  mission 
worked  well  in  most  cases  (and  was  most  cost  effective);  however,  some  integration  testing 
required  a  live  mission. 

-  Use  risk  reduction  tests  for  integration.  A  building  block  approach  was  used  successfully  to 
check  out  interfaces  at  the  lowest  level,  then  one  or  two  resources  at  a  time  were  added  to 
integrate  the  linked  configuration.  These  risk  reduction  tests  were  also  useful  for  developing 
analytical  tools. 

-  Use  a  stepped  approach  to  testing  where  each  successive  ADS  test  builds  on  the  success  of 
earlier  tests.  This  “test,  analyze,  fix,  test”  approach,  in  concert  with  structured,  independent 
testing  of  the  network,  will  greatly  improve  the  chances  for  successful  distributed  testing. 

-  Risk  reduction  testing  prior  to  actual  test  execution  provided  effective  rehearsals  and  was 
helpful  for  troubleshooting. 

-  The  FAT  and  FIT  series  of  federation  tests  were  invaluable  for  establishing  EW  Phase  2 
procedures  within  the  TCAC  and  with  AFEWES  and  ACETEF.  However,  personnel  were 
added  or  changed  locations  for  test  execution  which  impacted  test  rehearsal  learning.  The 
action  adopted  by  JADS  was  to  plan  appropriate  test  rehearsals  and  comprehensive 
integration  tests  for  Phase  3. 

-  Distributed  testing  requires  strong  systems  integration  and  systems  engineering.  There  are 
important  considerations  in  the  planning  process  that  are  not  well  understood  and  therefore 
may  lead  to  unanticipated  costs.  This  responsibility  is  difficult  to  manage  by  participants 
supplying  items  to  be  integrated.  If  the  sponsor  is  unable  to  provide  the  expertise  of  a 
systems  engineer  and  integrator,  an  independent  source  should  be  used.  Subject  matter 
experience  and  knowledge  of  computers  and  communications  technology  are  essential  for  the 
systems  integrator.  JADS  assumed  the  lead  systems  engineering  role  throughout  the  JADS 
EW  Test.  During  Phase  2  execution,  the  responsibility  of  system  engineering  unofficially 
transferred  to  other  IPT  members.  Quite  often,  IPT  members  were  also  responsible  for 
performing  development  tasks  and  delivery  of  several  key  software  elements.  This  made  it 
difficult  for  these  IPT  members  to  remain  unbiased  and  independent  during  integration.  The 
systems  engineer  needs  to  objectively  identify  and  aggressively  resolve  problems.  Using  an 
independent  systems  engineer  does  this  best. 

3.1.8  Data  Analysis 

-  There  are  two  unique  aspects  of  distributed  testing  that  affect  data  analysis  and  must  be 
planned  for  early.  The  first  is  the  requirement  to  do  real-time  data  analysis  to  support  test 
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control  and  test  conduct.  The  second  is  that  distributed  testing  generates  large  amounts  of 
data  in  a  short  time.  Without  careful  planning  and  testing  of  the  entire  data  collection, 
processing,  and  analysis  process,  the  analysts  will  be  either  hopelessly  lost  or  hopelessly 
buried  in  data. 

Test  design  should  include  as  much  real-time  analysis  as  possible.  This  will  increase  the 
success  rate  during  test  execution  and  minimize  time  required  to  repeat  test  activities  if  the 
equipment  problems  are  not  noticed  until  after  test  completion.  Future  testers  should  also 
consider  possible  actions  when  problems  are  discovered  during  test  execution.  The  decision 
to  stop  testing  until  the  problem  is  fixed  or  continue  and  account  for  the  problem  in  the 
results  is  a  difficult  decision  that  should  be  considered  long  before  test  execution  begins. 
During  the  various  test  phases,  real-time  analysis  became  more  and  more  crucial  to  the 
successful  execution  of  the  test.  Phase  3  was  largely  impacted  by  the  need  to  accomplish  as 
many  successful  runs  as  possible  in  the  least  amount  of  test  time.  Without  the  ability  to 
observe  and  critique  performance  from  the  various  federation  participants,  the  test  time 
would  have  been  lengthened  or  the  useable  test  data  collected  would  have  been  significantly 
decreased.  ADRS  and  site  observers  were  vital  to  correcting  operator  actions  and  clarifying 
the  rules  of  engagement.  SUT  observers  were  also  critical  in  determining  if  the  SUT  was 
performing  as  desired  as  the  test  was  executed.  Network  observers  were  also  needed  to 
observe  the  performance  of  network  equipment  during  test  execution.  The  ability  in  all 
phases  to  watch  threat  performance,  operator  performance,  and  SUT  performance  became  a 
cornerstone  to  successful  test  execution.  The  real-time  analysis  supplied  vital  information  to 
the  test  controller  who  could  ask  questions  about  specific  equipment  or  operators  as  soon  as 
problems  were  noticed.  During  Phase  3,  this  capability  corrected  severe  operator  training 
problems  at  AFEWES  and  SUT  problems  at  ACETEF  that  allowed  many  runs  to  be  saved  in 
the  final  data  sets. 

Effective  data  management  is  needed  as  ADS  can  generate  mountains  of  data.  A 
comprehensive  plan  will  clearly  identify  the  data  to  be  collected  at  each  site,  on-site 
processing  of  the  data,  and  data  to  be  transferred  to  the  analysis  center. 

Adequate  time  must  be  allowed  for  data  analysis  between  test  events.  Analysis  procedures 
should  be  rehearsed  to  better  understand  the  amount  of  time  needed  for  this  analysis. 

Analysts  should  become  very  familiar  with  the  analysis  software  products  very  early  in  the 
test  process.  If  possible,  products  should  be  chosen  that  automate  and  complete  the  most 
work  with  the  least  amount  of  intervention  and  modification  from  the  analysis  team.  If 
multiple  products  are  deemed  necessary,  the  amount  of  flexibility  in  each  application  to  read 
files  from  other  applications  is  very  important.  Some  consideration  should  be  given  to 
building  a  specific  process  that  will  accomplish  all  pieces  of  the  analysis  process  within  a 
single  application.  Training  analysts  on  the  selected  applications  should  also  be 
accomplished  early  in  the  analysis  process.  During  the  analysis  of  the  test  data,  the  lack  of 
integration  among  data  analysis  products  became  troublesome.  For  the  OAR  test,  the 
conversion  utilities  used  to  create  files  for  reduction  in  ADRS  from  the  OAR  data  files  were 
time  consuming.  Furthermore,  in  the  analysis  of  HITL,  Phase  2,  and  Phase  3  data,  the 
gathering  of  summary  statistics  and  the  execution  of  the  correlation  process  were  also  very 
time  consuming  because  the  entire  process  could  not  be  done  within  a  single  application. 
With  the  additional  work  needed  to  change  formats  of  data  among  the  various  pieces  of  the 
ADRS  software,  the  summary  statistics  and  graphical  representations  were  completed  using 
Microsoft®  Excel.  Exporting  the  data  to  Excel  was  time  consuming  in  itself  but  resulted  in 
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the  ability  to  modify  data  sets  and  greater  flexibility  in  creating  graphs  and  sorting  individual 
data  sets.  Further  work  was  required  to  perform  “correlation”  using  Statistix  because  Excel 
did  not  perform  the  Kolmogorov-Smimoff  test.  This  was  also  very  time  consuming  and 
■lengthened  the  time  needed  to  complete  the  analysis  process. 

Future  testers  wishing  to  perform  “correlation”  should  allow  for  engineering  assessments 
from  SMEs  to  be  able  to  obtain  useful  information  from  the  correlation  process.  The 
ultimate  question  between  data  sets  collected  from  two  different  test  phases  is,  "How  much 
difference  is  too  much  to  tolerate?"  The  EW  ‘^correlation”  process  provided  for  very  poor 
results  when  correlating  the  different  phases  of  the  EW  Test.  The  student’s  t  and  F  tests  used 
to  assess  data  sets  means  and  variances  were  very  rigid  and  provided  for  very  low  probable 
values  (P-values)  for  most  data  sets.  These  tests  examined  if  the  two  data  sets  considered 
came  from  exactly  the  same  population.  The  P-value  was  calculated  from  the  means, 
variances,  and  number  of  samples  in  each  data  set.  At  low  data  sample  counts,  these  tests 
were  more  flexible  in  determining  the  P-value  between  the  two  data  sets.  In  these  cases,  a 
small  difference  in  means  or  variances  did  not  always  generate  low  P-values.  However,  as 
sample  count  increased,  the  tests  became  more  and  more  rigid.  As  seen  in  the  J/S  correlation 
tables,  the  extremely  high  sample  counts  provided  for  P-values  of  .0000  when  the  data  sets 
graphically  aligned  quite  well.  The  Kolmogorov-Smimoff  test  was  not  quite  as  rigid  as  the  t 
and  F  tests,  but  the  results  were  almost  equally  poor.  This  can  lead  the  test  manager  into  a 
false  sense  of  failure  because  data  sets  between  test  phases  did  not  correlate.  Based  on  the 
requirements  of  the  “correlation”  process,  the  tests  could  be  relaxed  to  provide  more 
insightful  information  to  the  test  manager.  For  instance,  when  the  mean  miss  distances  for 
missile  shots  for  a  threat  system  are  24.0  and  25.5  feet  between  two  test  phases,  the 
correlation  tests  gave  moderately  low  P-values.  However,  the  sets  should  be  considered 
equal  if  the  blast  radius  of  the  missile  is  30  feet.  Engineering  assessments  were  needed  to 
determine  how  much  difference  between  two  data  sets  was  acceptable  before  the  data  sets 
were  not  considered  to  be  from  the  same  population.  The  current  process  did  not  provide 
meaningful  insight  into  the  threat,  SUT,  or  operator  performance,  nor  did  it  allow  the  tester  to 
assess  if  ADS  affected  the  MOP  results. 

System  data  should  be  validated  as  early  in  the  test  process  as  possible.  Validity  was  the 
lesser  of  the  two  problems.  Repeatability  should  be  checked  during  the  analysis  of  each 
phase  of  test  data.  If  repeatability  cannot  be  guaranteed  and  proven,  more  data  should  be 
collected,  if  possible,  or  the  analysis  reports  should  reflect  the  non-repeatable  nature  of  the 
data  sets.  When  ROE  or  more  samples  can  not  make  the  data  become  repeatable,  the 
correlation  process  should  not  be  used  on  that  particular  data  set.  The  “correlation”  process 
assumes  the  collected  data  from  the  different  test  phases  are  valid  and  repeatable.  If  the  data 
were  invalid,  they  were  not  useful  to  the  test  manager  to  judge  SUT,  threat,  or  operator 
performance.  If  the  data  were  not  repeatable,  the  “correlation”  of  such  data  ran  the  risk  of 
obtaining  both  false  positive  or  false  negative  “correlation”  simply  because  of  the  luck  of  the 
draw  with  the  collected  data.  Post-test  analysis  revealed  that  not  all  the  data  collected  were 
repeatable.  The  validation  process  only  asserted  validation  by  the  participating  agencies 
without  explicitly  checking  the  collected  data  by  subject  matter  experts.  Both  of  these 
problems  affected  the  poor  results  of  correlation  between  the  different  test  phases.  Because 
repeatability  and  validity  were  assumed  pretest  and  not  explicitly  checked,  the  results  of  the 
correlation  process  should  not  be  used  in  the  assessment  of  system  performance.  This  was 
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only  one  of  many  factors  leading  to  the  discredit  of  the  correlation  process  and  calling  for 
modifications  to  future  tests  where  correlation  is  used. 

MOP  definitions  should  be  modified  in  future  tests  to  assess  fewer  components,  or  the  data 
should  be  collected  in  a  manner  that  allows  the  analysis  team  to  better  determine  the 
individual  effects  of  each  source  of  variance.  Without  the  ability  to  perform  this  function,  it 
will  be  troublesome  to  make  definitive  and  valid  statements  about  the  ADS  effects  on  EW 
testing.  During  the  analysis  process,  it  became  very  difficult  to  assess  the  individual  variance 
sources  in  the  MOP  data  sets.  Operator  variance,  system  performance,  and  ADS 
performance,  for  instance,  affected  missile  miss  distance.  Determining  which  of  these 
sources  caused  the  largest  amount  of  variance  was  difficult  to  find  and  almost  impossible  to 
assert.  Other  MOPs,  such  as  correct  threat  identification  (ID),  response  time  and  correct 
ECM  technique  response  time,  were  largely  affected  by  data  latency.  However,  without  the 
data  sets  being  collected  in  two  different  manners  (both  with  and  without  latency  included),  it 
was  very  difficult  to  determine  if  the  lack  of  “correlation”  was  due  to  data  latency  or  system 
performance.  Many  other  instances  were  available  that  showed  the  mixture  of  different 
sources  of  variance  and  their  convoluted  influence  on  the  collected  data  sets.  It  was  nearly 
impossible  to  point  directly  to  the  source  of  variance  for  many  MOPs.  This  diminished 
JADS  ability  to  determine  the  effects  of  ADS  on  EW  testing.  Because  the  MOPs  were 
modified  to  assess  multiple  components  of  the  test  (e.g.,  ADS,  threat,  SUT,  and  operator 
performance)  the  ability  to  comment  on  the  specific  effects  from  each  component  was  greatly 
diminished. 

For  future  tests,  non-ADS  effects  should  be  understood  well  before  test  execution  in  order  to 
assess  each  component.  Modifications  should  be  made  to  ROE,  MOP  definitions,  or  the 
engagement  scenario  to  better  control  the  non-ADS  effects  or  to  at  least  be  more  able  to 
separate  the  effects  of  the  different  sources  of  variance.  Without  the  separation  and  control 
of  the  ADS  and  non-ADS  effects,  correlation  between  data  sets  may  still  prove  to  be  an 
impossible  task.  Based  on  the  analysis  performed  on  the  test  data,  it  was  discovered  that  non- 
ADS  effects  caused  the  largest  amount  of  variance  in  most  of  the  MOPs.  Operator  variance 
was  the  largest  source  of  variance  by  far.  Because  the  operator's  choices  on  when  to  switch 
modes,  how  well  to  track,  when  to  fire  missiles,  etc.,  affected  the  test  data,  constraining  this 
was  a  very  difficult  problem.  Even  among  the  expert  operators,  variance  from  the  operator 
was  still  larger  than  any  variance  caused  by  ADS  effects  such  as  data  latency,  data  loss,  data 
corruption,  etc.  Furthermore,  threat  differences  and  SUT  differences  among  the  different  test 
phases  also  contributed  to  the  variance  in  the  data  sets.  Without  constraining  the  non-ADS 
effects  on  the  MOPs  collected,  it  was  quite  difficult  to  determine  the  ADS  effects  on  the 
MOP  data.  Since  the  primary  objective  of  the  EW  Test  was  to  assess  the  ADS  impacts  to 
EW  testing,  the  success  at  this  project  were  mostly  qualitative  results  based  on  the 
understanding  of  the  MOP  definitions  and  the  qualitative  results  seen  through  the  various  test 
phases. 

Future  testers  should  attempt  preliminary  testing  to  determine  if  the  MOP  definitions  selected 
allow  the  expected  results  to  be  collected  from  the  test  execution.  More  so,  the  determination 
of  which  components  will  be  assessed  should  be  determined  very  early,  and  the  test  design 
should  be  modified  to  accommodate  these  assessments.  The  MOP/MOE  definitions  used  in 
current  EW  testing  allowed  for  many  various  effects  to  be  combined  into  a  single  measure. 
Missile  miss  distance  combined  the  effects  of  SUT,  threat,  operator,  and  ADS  performance, 
which  made  it  nearly  impossible  to  determine  the  individual  effect  of  ADS  impacts.  Other 
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MOPs,  such  as  correct  threat  ID  response  time  and  correct  ECM  technique  selection  response 
time  allowed  the  ADS  effects  to  be  quantified  but  only  if  the  data  were  collected  so  the  ADS 
effects  could  be  explicitly  removed  from  each  data  sample.  Most  other  MOPs  did  not  allow 
■for  an  accurate  assessment  of  ADS  impacts  to  EW  testing.  Without  the  ability  to  separate  the 
individual  effects  of  ADS  impacts,  the  successful  completion  of  the  JADS  tasking  to 
determine  the  utility  of  ADS  for  various  types  of  testing  was  weakened.  Because  the 
MOP/MOE  definitions  combined  the  different  sources  of  variance,  it  was  only  possible  to 
make  educated  guesses  about  the  impacts  of  ADS  to  EW  testing.  In  very  few  cases  was  it 
possible  to  effectively  determine  the  ADS  effects  on  the  performance  of  the  various 
components,  and  without  quantitative  data  to  back  JADS  claims,  the  results  and 
interpretations  of  ADS  utility  were  subject  to  individual  opinion. 

3.1.9  HLA 

-  HLA  was  adequate  to  support  the  EW  Test,  but  it  required  a  lot  of  tuning  and  trial  and  error 
testing  on  the  part  of  JADS  and  DMSO.  Better  documentation  that  provides  insight  into  the 
inner-workings  of  the  RTI  would  help.  Thorough  instrumentation  is  required  to  track  and 
isolate  problems  between  the  network  and  the  RTI.  The  T&E  community  needs  to  actively 
participate  in  the  Architecture  Management  Group  to  promote  the  development  of  HLA 
standards  and  products  that  meet  the  needs  of  the  T&E  community. 

-  When  JADS  began  working  with  the  RTI,  complete  documentation  on  the  correct  use  of  all 
the  RTI  services  and  calls  was  not  available.  JADS  was  surprised  to  learn  post-test  that  the 
reliable  distributor  servicing  the  federates  located  in  Albuquerque  was  incorrectly 
implemented.  The  following  is  a  detailed  discussion  of  the  reliable  distributor  and  how 
JADS  implemented  it  for  the  federates  in  Albuquerque.  Normally,  every  federate  includes  a 
reliable  distributor  (RELDISTR)  based  on  the  Internet  TCP,  since  the  RTI  best  effort 
communications  mechanism  provides  neither  guaranteed  delivery  to  all  message  recipients 
nor  in-order  message  delivery.  The  RELDISTR  is  used  to  send  reliable  data,  i.e.,  guaranteed, 
in-order  delivery  from  one  federate  to  one  or  more  other  federates.  During  the  analysis  of 
Phase  2  data  loss  and  data  delay  events,  there  were  many  instances  of  differential  latencies 
for  reliable  messages  sent  from  a  federate  on  one  test  node  to  two  or  more  federates  on  the 
other  nodes.  For  example,  a  latency-sensitive  jammer  technique  command  message  sent  by 
the  DSM  federate  at  ACETEF  might  arrive  with  a  normal  latency  at  AFEWES  and  two  of  the 
JADS  federates  but  be  delayed  to  the  other  two  JADS  federates  by  hundreds  of  milliseconds 
or  even  seconds.  When  DMSO  technical  support  was  queried  about  such  anomalies,  they 
advised  JADS  that  Phase  2  actually  had  three  RELDISTR  running  on  the  radio  frequency 
environment  (RFENV)  host  at  the  JADS  node  in  addition  to  single  RELDISTR  in  the 
federates  at  the  AFEWES  and  ACETEF  nodes.  In  an  effort  to  minimize  the  amount  of  traffic 
on  the  WAN,  the  DMSO  liaison  for  JADS  recommended  that  a  single  reliable  distributor  for 
the  federates  in  the  TCAC  be  used  during  Phase  3.  This  also  was  desirable  to  eliminate  some 
types  of  differential  latency  problems.  The  RFENV  federate  was  chosen  to  host  the  reliable 
distributor  for  the  TCAC.  The  RFENV  federate  had  to  be  started  first,  since  all  other 
federates  would  attempt  to  connect  to  its  RELDISTR.  Because  of  the  two  redundant 
RELDISTR  in  the  runtime  infrastructure  executive  (RTIEXEC)  and  federation  executive 
(FEDEX)  on  the  same  SGI  02  host,  redundant  TCP  connections  were  apparently  created 
(based  on  post-test  network  packet  sniffer  evidence)  between  the  RELDISTR  on  RFENV  and 
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those  at  APEWES  and  ACETEF.  The  extra  RELDISTR  and  the  redundant  network 
pathways  probably  were  the  cause  of  some  differential  latency  events  during  Phase  2.  The 
RTffiXEC  had  its  own  RELDISTR,  so  for  Phase  3  all  federates  in  the  TCAC  were 
configured  to  use  the  RTIEXEC  RELDISTR.  However,  because  of  a  problem  with  RTI 
Version  1.3  Release  5,  this  required  that  the  RTIEXEC  be  started  with  one  version  of  the 
RTLrid  file,  which  then  had  to  be  replaced  by  a  second  version  before  the  FEDEX  and  the 
RFENV  federate  were  started.  This  minor  inconvenience  was  handled  by  means  of  a  UNDC 
shell  script.  There  are  two  primary  implications,  to  ADS-based  tests.  First,  federations  with 
multiple  federates  on  a  local  area  network  (LAN)  should  consider  using  a  single  RELDISTR 
per  LAN.  Second,  RTI  developers  need  to  clearly  document  how  to  correctly  implement 
nondefault  configurations  so  that  federations  can  take  full  advantage  of  the  RTI  features. 
Further  implications  are  discussed  below.  Designers,  instrumenters,  and  executors  of  real¬ 
time,  performance  federations  with  latency-sensitive  messages  sent  via  the  RTI  reliable 
communications  protocol  to  two  or  more  federates  on  other  distributed  test  nodes  need  to 
carefully  consider  the  potential  consequences  of  differential  latencies.  That  is  because 
differential  latencies  can  cause  the  federates  to  have  different  perceptions  of  if,  and  when, 
critical  events  happened.  The  original  RTI  developer’s  decision  to  use  TCP  for  reliable 
traffic  may  have  unavoidable,  long-term,  negative  consequences  that  may  cause  trouble  for 
some  real-time,  performance-oriented  HLA-based  simulations.  For  example,  during  RTI 
performance  testing  leading  up  to  Phase  2,  JADS  learned  that  TCP  implementations  differ 
significantly,  not  only  among  those  of  different  vendors,  but  also  among  different  operating 
system  version  releases  from  the  same  vendor.  A  significant  example  of  this  is  in  the 
availability  of  the  so-called  TCP_NODELAY  option  that  would  allow  the  RELDISTR  s  TCP 
to  acknowledge  incoming  TCP  segments  without  delay.  This  option  was  not  available  in 
SGFs  IRIX  6.3  operating  system  but  was  available  in  IRIX  6.5  Sun  Solaris  and  some  other 
operating  systems.  Use  of  this  option  within  the  RTI  and  by  the  federate  developers  for  non- 
HLA  federate  components  (e.g.,  the  DSM  PC  software)  probably  would  have  reduced  the 
latencies  of  reliable  messages.  Also,  it  is  not  at  all  clear  that  RTI  developers  using  TCP  for 
reliable  distributor  implementations  have  any  means  to  guarantee  that  the  TCP  underlying  a 
transmitting  RELDISTR  sends  all  copies  of  a  reliable  message  intended  for  two  or  more 
recipients  with  minimal  delay  between  outbound  copy  over  a  separate  TCP  connection.  That 
is  because  the  TCP  protocol  was  never  developed  with  this  type  of  performance  requirement 
in  mind.  It  is  also  unclear  as  to  whether  intermediate  RELDISTRs  might  introduce 
additional  differential  latencies  because  of  a  lack  of  control  over  the  details  of  TCP  actions 
on  two  or  more  independent  TCP  connections  (a  TCP  connection  consists  of  two  pair  of  IP 
addresses  and  port  numbers,  one  for  the  source  and  one  for  the  destination). 

High  performance  federations  can’t  treat  the  RTI  as  a  black  box.  Just  because  federation 
designers  are  careful  about  which  federates  subscribe  to  data  (in  an  effort  to  reduce  WAN 
traffic)  doesn’t  mean  that  the  data  aren’t  being  sent  to  the  federate  anyway.  Federation 
designers  need  to  think  carefully  about  the  instmmentation  for  monitoring  their  federations, 
and  that  instrumentation  should  be  in  place  well  before  the  start  of  formal  integration  testing. 
Phase  2  showed  that  RTI  loggers,  DIS-style  passive  loggers,  Internet  ping  probing,  and 
network  error  printouts  provide,  at  best,  only  circumstantial  and  limited  evidence  to  diagnose 
the  root  causes  of  most  data  latency  and  data  loss  problems  for  HLA-based  distributed 
simulations.  RTI  developers  need  to  document  how  the  RTI  establishes  multicast  groups  so 
that  federation  designers  can  take  full  advantage  of  what  the  RTI  has  to  offer.  Details  on 
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how  the  RTI  handles  its  communications  are  deliberately  withheld  from  the  user.  This  is 
done  to  encourage  users  to  treat  the  RTI  as  a  black  box  and  adhere  to  the  interface 
specification.  This  works  for  most  users;  however,  T&E  users  have  a  need  to  know  how 
•communications  are  handled.  JADS  was  surprised  to  learn  post-test  how  the  RTI  really 
created  multicast  groups.  Instead  of  separate  multicast  groups  being  established  according  to 
actual  publish/subscribe  topology,  JADS  best  effort  data  were  sent  in  a  single  multicast  group 
to  which  all  federates  were  connected.  Each  local  instance  of  the  RTI  had  to  deal  with  all 
messages  even  if  its  federate  did  not  subscribe  to  all  messages.  This  should  have  been 
known  in  early  design  so  that  different  implementations  could  have  been  tested.  The 
following  is  a  detailed  discussion  on  how  this  worked  within  RTI  1.3  release  4  and  5.  Also, 
there  is  a  discussion  of  the  data  losses  that  were  apparent  and  how  the  multicast 
implementation  may  have  contributed.  When  the  RTIEXEC  starts  execution,  it  transmits 
Internet  Group  Management  Protocol  (IGMP)  “report”  messages  to  join  several  IP  multicast 
groups  which,  for  the  JADS  federation,  had  Class  D  Internet  addresses  of  the  form 
224.253. XXX. yyy.  The  HEDEX  does  the  same  when  it  begins,  and  so  does  each  federate  as  it 
Joins  the  federation.  These  IP  multicast  groups  provide,  via  the  UDP,  the  RTFs  one-to-one 
and  one-to-many  best  effort  communications  infrastructure.  The  RTI  within  each  federate 
uses  the  stream  map  in  the  RID  file  (i.e.,  the  file  RTI.rid)  to  determine  to  which  multicast 
group  a  particular  type  of  best  effort  message  should  be  sent  to  reach  a  specific  federate  or 
group  of  federates.  The  specific  multicast  group  joined  by  a  federate  depends  on  when  it 
joins  versus  the  other  federates.  Also,  as  new  federates  join  (or  joined  federates  resign),  the 
RTI  dynamically  redirects  best  effort  traffic  within  the  established  multicast  groups.  After 
Phase  2,  JADS  discovered  this  behavior  by  using  network  packet  sniffers  on  the  SGI  02 
hosts  and  eventually  learned  from  the  RTI  developer  that  the  stream  map  in  the  RID  file 
provided  to  JADS  caused  all  federates  joining  after  the  third  federate  to  stop  joining  new 
multicast  groups  in  addition  to  those  already  created.  Instead,  they  joined  a  broadcast 
multicast  group  (224.253.1.0),  and  federation  traffic  formerly  sent  to  specific  multicast 
groups  was  redirected  to  that  group.  The  result  was  that  all  federates  received,  even  if  they 
did  not  subscribe,  almost  all  best  effort  messages,  and  the  local  RTI  component  (LRC)  within 
the  federates  had  to  process  and  discard  those  unwanted  messages.  Thus,  for  example,  the 
LRC  in  the  hand-off  federate,  which  did  not  subscribe  to  threat  performance,  had  to  receive, 
process,  and  discard  five  20-hertz  message  streams  from  the  platform  and  AFEWES 
federates.  During  Phase  2,  there  were  many  instances  of  best  effort  data  losses  that  were 
unusual  in  two  ways:  they  were  one-way  losses,  meaning  that  messages  between  two  or  more 
federates  were  lost  in  one  direction  but  not  in  the  opposite  direction;  and  they  were  selective 
losses,  e.g.,  the  DSM  federate  did  not  receive  link  health,  live  entity  state,  and  threat 
performance  messages  from  the  platform  federate  at  JADS,  but  did  receive  link  health 
messages  from  the  other  three  JADS  federates.  These  losses  cannot  be  explained  by  network 
problems  such  as  a  short  outage  on  one  of  the  T-1  lines,  loss  of  crypto  synchronization,  etc., 
since  those  problems  would  affect  all  best  effort  traffic  in  both  directions  between  two  test 
nodes.  This  suggested  that  these  selective,  one-way  best  effort  data  losses  might  have  been 
due  to  some  problem  with  the  RTFs  use  of  IP  multicast  groups.  Or,  they  might  have  been 
caused  by  “pruning”  of  some  IP  multicast  addresses  by  the  protocol  independent  multicast- 
dense  mode  (PIM-DM)  routing  protocol  that  the  JADS  routers  used.  Because  of  lack  of 
adequate  documentation  for  the  RTI  RID  file,  JADS  unknowingly  used  a  RID  file  with  a 
stream  map  that  was  probably  not  appropriate  for  a  federation  with  six  or  seven  federates. 
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As  a  result,  almost  all  our  best  effort  data  were  sent  to  all  federates  unnecessmly  loading 
some  of  them.  Perhaps  because  of  IP  multicast-related  bugs  in  RTI  Version  1.3  Release  4, 
and/or  router  protocol  pmning  of  RTI  IP  multicast  addresses,  JADS  expenenced  many 
unusual,  selective,  one-way  best  effort  data  loss  events.  For  runs  36  and  107,  these  even  s 
had  consequences  that  caused  unacceptable  response  times  for  some  DSM  jammer  technique 
commands.  The  RTLrid  file  could  be  modified  with  a  new  stream  map  to  provide  more 
multicast  groups  to  the  federation.  The  RTI  developer’s  suggestion  of  using  data  distribution 
management  could  have  been  accepted  but  was  rejected  for  Phase  3  for  the  same  reason  that 
it  was  not  used  for  Phase  2;  namely,  there  appeared  to  be  a  significant  risk  of  adding 
unacceptable  latencies  to  the  real-time,  performance-oriented  federation.  Regarding  t  e 
second  problem,  PC-based  network  packet  sniffers  on  the  LANs  leading  to  the  routers  were 
added  at  all  three  nodes  to  improve  the  instrumentation. 

Federations  should  experiment  with  different  transport  modes  to  determine  the  optimum  mix 
of  transport  modes.  RTI  developers  should  have  tools  or  performance  me^urements  to 
guide  federation  developers  as  they  design  and  integrate  their  architectures.  TSPI  data  losses 
among  platform  federate,  the  DSM,  AFEWES,  and  TCF  occurred  frequently  tog 
federation  integration.  JADS  and  DMSO  investigated  this  problem.  A  work-around  solution 
was  found.  This  solution  was  a  fixed  join  process  for  federates  prior  to  each  test  run.  ^ter 
the  solution  was  implemented,  data  losses  among  the  DSM,  AFEWES,  and  the  platform 
federate  were  still  observed.  These  losses  manifested  themselves  m  the  apparent  hovering 
aircraft  observed  in  the  FIT.  It  was  not  clear  what  caused  the  data  loss,  but  dead  reckoning 
aircraft  position  provided  an  acceptable  solution.  This  data  loss  coupled  with  the  dead 
reckoning  implementation  at  AFEWES  was  the  suspected  cause  of  an  extremely  large  imss 
distance  value  on  one  missile  shot.  RTI  bundling  of  federate  data  for  transmission  ma  e 
troubleshooting  data  flow  and  transmission  problems  more  difficult.  Our  tools  assessed 
hardware  performance  only.  Instrumentation  for  federation  performance  evaluation  used  in 
Phase  2  was  inadequate.  It  lacked  the  ability  to  examine  data  passed  between  RTI  instances. 
Best  effort  data  could  be  dropped  by  the  network  without  notification  or  without  any  fau  s 
reported  by  the  hardware.  Phase  3  network  instrumentation  must  be  expanded  to  include 
network  sniffers  to  monitor  network  traffic  between  the  sites.  Federation  performance  v^ed 
as  the  mix  of  reliable  and  best  effort  data  changed.  Through  trial  and  error  fewer  problems 
with  latency  and  data  loss  were  noted  if  less  reliable  traffic  was  published  within  t  e 
federation.  However,  this  was  subjective  because  no  tools  were  available  to  test  me 
performance  envelope  of  the  architecture.  Federation  performance  was  poor  when  integration 
testing  started.  The  link  health  messages  were  required  to  be  published  as  best  ettort 
messages  to  correct  this  problem.  This  change  was  made  late  in  the  integration  effort  to 
further  tune  the  architecture  with  the  real  federates.  To  correct  for  these  problems  LHC 
messages  were  published  best  effort  during  both  ADS  testing  phases. 


96 


3.2  Programmatic 

3.2;1  Procedures 

-  The  number  and  types  of  procedural  lessons  learned  JADS  identified  clearly  substantiate  a 
necessity  to  develop  standards  for  the  use  of  ADS. 

-  Network  management  and  troubleshooting  must  be  disciplined  and  organized  with  a 
thorough  understanding  and  strong  configuration  control  of  the  distributed  testing  network. 
This  approach  enabled  the  ETE  Test  Phase  2  team  to  quickly  recover  from  initial  pretest 
network  difficulties  and  to  go  on  and  achieve  five  days  of  successful  execution  and  data 
collection. 

-  Flexibility  is  also  needed.  When  one  of  the  ETE  Test  network’s  T-1  lines  was  disconnected, 
JADS  personnel  were  able  to  quickly  develop  and  implement  a  contingency  plan.  Upon 
restoration  of  the  T-1  line,  the  network  was  soon  returned  to  its  original  configuration  and  the 
test  continued. 

-  Quantitative  validation  has  limitations.  JADS  intent  was  to  quantitatively  verify  missile 
simulation  performance  against  live  fire  data.  However,  as  only  one  live  fly  event  was 
available  to  support  the  process,  a  modified  approach  including  both  quantitative  and 
qualitative  methods  was  used  and  successfully  identified  invalid  results. 

-  The  requirements  for  a  distributed  test  must  be  clearly  defined  early  in  the  planning  phase. 
This  includes  user  requirements,  support  agency’s  stated  actions,  and  operations  security 
requirements.  Planning  and  coordination  details  will  be  much  more  involved  than  in  a 
traditional,  nondistributed  test. 

-  Existing  range  procedures  had  to  be  modified  for  ADS.  The  existing  test  procedures  were 
only  written  for  individual  facilities,  so  a  new  combined  checklist  was  created  for  distributed 
testing  applications. 

-  Get  SUT  experts  involved  from  the  beginning. 

-  Additional  time  is  needed  before  the  beginning  and  after  the  end  of  each  testing  period.  One 
hour  is  recommended  for  setup,  and  two  hours  at  the  end  for  data  logging,  data  archiving, 
data  transfer,  and  laboratory  reclassification. 

-  Briefings  are  needed  before  and  after  each  mission. 

-  Configuration  control  is  essential.  This  one  obvious  area  was  one  of  great  challenge 
considering  the  many  sites  involved  and  the  multiple  uses  of  each  site. 

-  The  requirements  for  a  distributed  test  must  be  clearly  defined  early  in  the  planning  phase. 
Detailed  planning  and  coordination  are  required  to  ensure  a  common  understanding  of  all 
requirements,  procedures,  and  test  objectives  since  individual  facilities  are  generally 
unfamiliar  with  conducting  coordinated,  distributed  T&E  tests. 

-  Flexibility  in  planning  is  essential.  When  doing  something  that  has  never  been  done  before, 
preconceived  notions  of  how  the  test  should  be  executed  will  have  to  change  as  more  is 
learned.  Be  open  to  new  ideas,  as  some  of  the  old  ideas  from  the  early  stages  of  a  distributed 
test  program  may  become  very  expensive  to  bring  to  fruition.  The  ETE  Test  Phase  2  was 
originally  slated  to  have  nine  scenarios.  As  the  requirements  for  each  scenario  increased, 
their  development  costs  also  grew.  These  added  costs  eventually  led  to  the  deletion  of  the 
last  four  scenarios. 
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Plan  for  the  unexpected.  Halfway  through  the  ETE  Test  Phase  2  one  of  the  key  T-1  lines  was 
inadvertently  disconnected.  This  delayed  the  test  by  two  days  during  the  most  critical 
portion  of  the  test  while  technicians  restored  the  lines.  Travel  plans  had  to  be  changed  and 
budgets  were  strained.  If  possible,  plan  for  extra  days  on  the  end  of  a  test  period  that  can  be 
eliminated  if  all  goes  well.  It  is  much  easier  to  return  early  than  to  stay  longer. 

Minimize  the  impact  of  hardware  problems.  When  using  a  complicated  distributed  testing 
network  with  a  vast  array  of  equipment,  hardware  problems  will  occur.  Plan  in  such  a  way 
that  unexpected  hardware  problems  do  not  completely  disrupt  the  test.  During  the  ETE  Test 
Phase  2,  steps  were  taken  to  ensure  that  hardware  problems  did  not  disrupt  the  test  for  long 
periods.  For  example,  data  saves  were  accomplished  frequently.  In  addition,  the  network 
was  constantly  monitored  to  ensure  that  hardware  problems  were  fixed  as  soon  as  possible. 
Use  a  stepping  stone  approach  to  testing  where  each  successive  distributed  test  builds  on  the 
success  of  earlier  tests.  This  “test,  analyze,  fix,  test”  approach,  in  concert  with  structured, 
independent  testing  of  the  network,  will  greatly  improve  the  chances  for  successful 

distributed  testing.  •  f  ^ 

Risk  reduction  testing  prior  to  actual  test  execution  will  help  test  team  personnel  identify  an 

resolve  potential  distributed  testing  system  problems. 

Understand  communication  requirements.  Because  of  some  changes  in  the  test,  voice 
communication  requirements  between  the  Fort  Hood  node  and  the  White  Sands  node  were 
dramatically  increased.  The  only  way  for  those  nodes  to  communicate  was  through  the  direct 
line  to  the  TCAC.  This  tied  up  the  direct  line  for  extended  periods.  For  the  next  test,  another 
connection  was  added  for  direct  communications  between  Fort  Hood  and  White  Sands. 
Briefings  are  needed  before  and  after  each  distributed  test.  These  briefings  should  include 
such  information  as  the  test  objectives,  telephone  numbers  to  use  for  test  control,  the  test 
configuration  of  each  facility,  instrumentation  and  data  collection  requirements,  go/no  go 
criteria,  contingency  and  backup  plans,  and  test  conduct.  A  briefing  checklist  should  be 
developed  and  used. 

Test  control  procedures  should  be  well  rehearsed.  When  many  people  are  communicating  on 
one  phone  line,  a  response  order  should  be  established  and  strictly  followed  to  save  valuable 
test  time. 

Take  advantage  of  the  opportunities  provided  by  distributed  testing  technology.  At  the 
beginning  of  Phase  2,  the  ETE  Test  team  intended  to  log  all  operator  actions  in  the  LGSM. 
As  the  test  progressed,  the  JADS  analysts  realized  obtaining  data  from  the  LGSM  operators 
would  hamper  their  activities.  Making  use  of  the  increased  test  time  provided  by  the 
distributed  testing  environment,  the  JADS  analysts  were  able  to  automate  most  of  their 
LGSM  data  collection  activities  and  reduce  the  impact  on  the  LGSM  operators. 

Effective  data  management  is  needed.  Linked  facilities  can  generate  a  large  volume  of  data 
at  distributed  locations.  Without  careful  planning,  key  data  may  not  be  collected  an^or 
transmitted  to  the  analysis  center,  and  data  collected  at  the  network  nodes  may  not  be  in  a 
useful  form  for  centralized  analysis.  Before  distributed  testing,  a  comprehensive  data 
management  plan  must  clearly  identify  the  data  to  be  collected  at  each  network  node,  on-site 
processing  of  the  data,  and  the  data  to  be  transmitted  to  the  analysis  center. 

Pretest  rehearsals  are  also  useful  for  improving  evaluation  procedures.  The  ETE  Test  team 
improved  its  data  collection  and  analysis  processes  as  a  result  of  experiences  from  the 
functionality  and  integration  and  risk  reduction  tests. 
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Have  a  centralized  test  control  center.  The  JADS  TCAC  was  configured  to  allow  for 
convenient,  instant  communications  with  all  the  nodes.  It  acted  as  the  central  point  of 
contact  between  the  nodes  and  for  all  problems.  The  test  controller  kept  track  of  test 
.  progress  and  documented  any  problems  that  occurred. 

Establish  control  over  resources.  Linking  various  facilities  using  distributed  testing  can 
require  significant  development  of  facility  interface  hardware  and  software.  A  well- 
organized  approach  by  experienced  personnel  enabled  the  ETE  Test  team  to  surmount 
problems  encountered  at  Fort  Hood,  the  most  complicated  node  in  terms  of  getting  all 
necessary  hardware  established  and  connected  before  Phase  2. 

Time  sources  must  be  synchronized  using  a  “master  clock”  and  then  validated  at  each 
network  node. 

Changes  and  upgrades  to  aircraft  instrumentation  delayed  development.  Specially 
instrumented  aircraft  were  required  to  support  the  LFP  flights.  Because  of  the  small  number 
of  such  aircraft,  the  LFP  schedule  was  very  sensitive  to  periodic  aircraft  phase  inspections, 
software  upgrades,  and  higher  priority  missions. 

Laboratory  replays  served  as  an  excellent  method  of  test  rehearsal. 

ADS  test  requirements  must  be  clearly  defined  early  in  the  test  planning  phase,  since 
individual  facilities  are  generally  unfamiliar  with  conducting  coordinated,  distributed  tests. 
Detailed  planning  for  run  management  is  necessary  before  testing  commences.  During  daily 
testing,  a  run  matrix  was  used  to  determine  which  runs  were  executed;  what  procedures  were 
used  to  start  the  federates,  models  and  simulators;  and  to  initiate  the  run,  stop  the  run  and 
shut  down  of  the  federates.  The  work  done  during  preliminary  testing  (e.g.,  FAT,  FIT) 
provided  a  repeatable  methodology  for  orderly  federation  operation  that  was  used  for  the 
formal  test.  Nonetheless,  problems  were  frequently  encountered,  errors  were  made,  and 
unanticipated  issues  arose. 

JADS  was  highly  dependent  upon  time  synchronization  of  all  federate  computers  and 
software.  Any  requirement  for  synchronization  requires  the  ability  to  verify  that  the 
requirement  is  being  met.  For  example,  if  one  millisecond  synchronization  accuracy  is 
required,  then  a  capability  to  measure  time  between  two  computers  at  one  millisecond 
precision  is  necessary.  However,  software  tools  were  not  available  to  measure  accuracy  at 
that  level.  In  fact,  testing  time  synchronization  across  the  federation  was  more  art  than 
science.  Even  with  time  synchronization  and  the  time  cards  implemented  in  all  computers, 
instances  were  noted  where  time  synchronization  “slipped”  affecting  latency  measurements. 
A  few  occurrences  of  time  synchronization  problems  across  the  federation  were  observed 
requiring  JADS  to  research  particular  runs  after  daily  testing.  JADS  consistently  followed 
documented  time  synchronization  procedures  and  hardware  settings.  Site  support  personnel 
were  relied  upon  to  implement  procedures  and  verify  settings  daily. 

Configuration  changes  on  tools  (analysis  federate,  ADRS  display,  DSM  joining  process, 
AFEWES  dead  reckoning  algorithms)  could  have  severe  impacts.  Configuration  changes, 
even  seemingly  trivial  ones,  must  be  coordinated  at  all  levels.  JADS  stressed  configuration 
management  procedures  for  Phase  3  and  enforced  their  use. 

Operator  boredom  with  the  repetitiveness  of  test  mns  at  AFEWES  may  have  contributed  to 
afternoon  run  differences.  JADS  attempted  to  minimize  turn-around  time  between  runs.  The 
time  between  runs  from  start  time  to  start  time  was  usually  between  6  and  8  minutes. 
Frequently,  the  government  only  knows  in  general  terms  what  is  needed  to  execute  tests  in  a 
geographically  distributed  environment.  Test  design  must  mature  to  identify  the  specific 
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capabilities  that  each  facility  will  provide  before  specifications  are  created.  This  generally 
precludes  creating  good  performance  specifications  prior  to  contract  award.  Sufficient 
tasking  must  be  included  in  the  SOW  to  ensure  that  government  interests  are  covered  and  the 
lead  from  the  contractor  has  a  leverage  tool  to  ensure  the  work  is  executed  on  time  with  good 
quality.  Sequential  contract  awards  may  be  used  to  mitigate  risks  associated  with  loose 
SOWS.  For  JADS  EW  abbreviated  SOWs  and  reduced  deliverables  resulted  in  differences  in 
expectations  between  contractors  and  the  government.  The  loosely  defined  SOW  allowed 
the  analysis  team  to  continue  to  refine  software  tequirements  for  a  critical  piece  of  software 
necessary  for  the  test  execution  well  beyond  the  date  it  should  have  been  finalized.  Several 
MOPs  were  modified  from  the  non-traditional  calculation  to  help  measure  ADS  effects.  This 
proved  more  difficult  than  expected.  Since  delivery  schedules  were  not  clearly  defined,  the 
contractor  permitted  these  discussions  to  go  on  well  beyond  the  time  needed  to  code  and  test 
the  software  to  meet  the  government’s  expected  delivery  date.  All  parties  were  trying  to  get 
the  best  insight  into  ADS  effects  on  the  EW  Test  measures  while  balancing  impacts  to  the 
software.  The  problem  was  resolved  when  the  government  program  manager  froze  software 
requirements  and  provided  the  contractor  a  specific  delivery  date.  A  second  impact  was 
related  to  the  level  of  on-site  test  support.  The  loosely  defined  SOW  allowed  the  contractor 
to  reallocate  on-site  resources  earlier  in  the  test  design  to  support  other  test  activities.  The 
reallocation  was  discussed  with  the  government;  however,  the  impact  to  on-site  support 
during  Phase  2  was  not  explicitly  negotiated.  As  a  result,  the  government  received  less 
support  than  expected.  Once  the  software  requirements  were  frozen,  the  updated  software 
was  delivered  in  time  for  test  execution. 

Schedule  may  be  the  hardest  factor  in  ADS  testing  to  control  because  it  is  influenced  by  both 
internal  factors  (e.g.,  the  ability  of  the  different  facilities  to  work  together  to  identify  and 
solve  problems  quickly)  as  well  as  external  factors  (e.g.,  other  tests  the  facility  must  support 
and  how  much  influence  those  tests  have).  Aggressive  management  of  all  development 
efforts  and  deliverables,  effective  risk  management,  and  starting  the  effort  with  enough  cost, 
schedule,  and  performance  trade  space  are  all  essential  ingredients  to  successful  test 
execution.  EW  tests  require  several  critical  assets  to  execute  successfully.  Delays  in  one  of 
these  critical  assets  impact  the  overall  test  schedule.  This  is  a  larger  problem  with  ADS  since 
delays  require  rescheduling  multiple  facilities,  each  with  unique  time  and  asset  constraints. 
The  EW  Phase  2  test  schedule  slipped  because  of  delays  in  obtaining  data  from  the  EW  Test 
Phase  1 .  The  DSM  required  response  time  data  from  the  system  integration  laboratory  (SIL) 
test  for  calibration.  The  first  two  attempts  to  collect  these  data  at  the  OAR  and  HTTL  tests 
failed  causing  the  need  to  perform  the  SIL  test.  Because  of  these  previous  failures,  the 
response  time  data  were  collected  much  later  than  required  to  prepare  for  Phase  2  test 
execution.  As  a  result,  this  test  phase  was  delayed  to  properly  calibrate  the  DSM  prior  to  test 
execution.  JADS  became  more  aggressive  in  managing  the  schedule  and  working  with  the 
supporting  organizations  to  ensure  that  resources  were  ready  and  in  place  to  support  the  test 
as  scheduled.  JADS  also  stated  the  test  was  not  executable  if  it  slipped  again.  No 
organization  wanted  to  be  responsible  for  canceling  the  test  event,  so  extra  efforts  were  made 
by  everyone  to  ensure  the  test  executed  successfully. 

Perhaps  the  most  important  lesson  learned  from  Phase  2  and  the  preparations  for  it  was  the 
critical  importance  of  careful  planning  and  preparations  at  the  earliest  stages  of  the  program. 
It  is  better  to  avoid  problems,  since  there  may  not  be  enough  time  and/or  money  to  find  and 
fix  them  later.  This  seems  especially  true  for  ADS  programs.  The  nature  of  ADS  brings 
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multiple  facilities  together,  each  having  their  own  development  style  and  practices  and  each 
bringing  a  potentially  different  understanding  of  the  problem.  This  is  very  similar  to  having 
multiple  facilities  working  together  to  develop  a  single  software  package.  Any  actions  that 
•reduce  ambiguity  in  the  interface  design  will  reduce  the  risk  of  the  program.  This  is  very 
important  for  ADS-based  tests  since  it  may  be  difficult  to  slip  test  schedules  when  multiple 
facilities  are  involved.  Hence,  the  importance  of  a  good  ICD  and  enforcing  the  same 
methods  of  compliance  from  the  start  of  software  development.  An  ICD  was  developed  for 
the  JADS  federation  to  guide  software  developer.S.  Two  problems  were  identified  relating  to 
this  ICD:  (1)  nonconformance  to  the  ICD  and  (2)  differences  in  interpretation  of  complex 
concepts.  Prior  to  the  test,  the  description  of  the  coordinate  transformation  was  agreed  to  be 
acceptable  by  all  participants;  however,  facilities  developed  different  implementations  of  the 
software  when  coding  was  finished.  The  problem  was  finally  resolved  when  JADS  provided 
sample  transformation  pairs  for  testing  each  facility’s  algorithm.  These  sample  data  points 
should  have  been  included  in  the  JADS  ICD  to  avoid  confusion.  In  some  instances  software 
was  developed  that  did  not  conform  to  the  ICD.  Because  of  the  lack  of  detailed  acceptance 
testing  these  nonconformance  problems  were  not  found  until  very  late  in  the  integration 
process.  As  a  result,  decisions  had  to  be  made  either  to  bring  the  software  into  conformance 
or  to  change  the  ICD  in  order  to  maintain  test  schedule.  For  example,  problems  with  the 
federate  message  sequence  numbers  illustrate  both  the  test  and  post-test  impacts.  For  each 
instance  of  a  simulation  object,  the  federates  should  have  used  sequence  numbers  in  outgoing 
messages  starting  from  1  and  incremented  by  1  for  each  successive  message.  However, 
because  of  a  combination  of  ambiguous  ICD  wording  and  lack  of  early  ICD  compliance 
testing  and  enforcement,  the  TTH  and  AFEWES  federates  transmitted  message  sequences 
that  did  not  conform  to  the  same  sequence  numbering  scheme.  During  Phase  2  test  execution 
this  became  a  problem  with  the  DSM  PC’s  real-time  error  checking  for  incoming  SMC 
messages.  The  sequence  number  was  used  by  the  DSM  to  detect  missing  and  out-of-order 
messages.  Since  the  sequence  numbers  were  not  set  correctly,  the  error  reports  were 
misleading  and  ineffective.  During  the  post-test  analysis,  improper  message  sequence 
numbers  for  several  message  types  made  it  more  difficult  to  detect  and  analyze  runs  with 
data  loss  and  latency  problems  for  the  ADS  analysis  process.  In  particular,  it  greatly 
complicated  the  calculation  of  overall  latencies  for  the  critical  combination  of  outgoing  SMC 
messages  and  the  corresponding  jammer  technique  command  messages  generated  by  the 
DSM.  To  correct  for  these  problems  message  sequence  counters  were  corrected  for  both  the 
TTH  and  AP^WES  threat  federates  for  Phase  3.  Wording  in  the  ICD  was  changed  to  be  less 
ambiguous. 

For  future  tests,  the  non-ADS  effects  should  be  understood  well  before  test  execution  in 
order  to  assess  each  component.  Modifications  should  be  made  to  rules  of  engagement, 
MOP  definitions,  or  the  engagement  scenario  to  better  control  the  non-ADS  effects  or  to  at 
least  be  more  able  to  separate  the  effects  of  the  different  sources  of  variance.  Without  the 
separation  and  control  of  the  ADS  and  non-ADS  effects,  correlation  between  data  sets  may 
still  prove  to  be  an  impossible  task.  Based  on  the  analysis  performed  on  the  test  data,  it  was 
discovered  that  non-ADS  effects  caused  the  largest  amount  of  variance  in  most  of  the  MOPs. 
Operator  variance  was  the  largest  source  of  variance  by  far.  Because  the  operator's  choices 
on  when  to  switch  modes,  how  well  to  track,  when  to  fire  missiles,  etc.,  affected  the  test  data, 
constraining  this  was  a  very  difficult  problem.  Even  among  the  expert  operators,  variance 
from  the  operator  was  still  larger  than  any  variance  caused  by  ADS  effects  such  as  data 
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latency,  data  loss,  data  corruption,  etc.  Furthermore,  threat  and  SUT  differences  among  the 
different  test  phases  also  contributed  to  the  variance  in  the  data  sets.  Without  constraining 
the  non-ADS  effects  on  the  MOPs  collected,  it  was  quite  difficult  to  determine  the  ADS 
effects  on  the  MOP  data.  Since  the  primary  objective  of  the  EW  Test  was  to  assess  the  ADS 
impacts  to  EW  testing,  the  success  at  this  project  were  mostly  qualitative  results  based  on  the 
understanding  of  the  MOP  definitions  and  the  qualitative  results  seen  through  the  various  test 

phases. 


3.2.2  Costs 

-  The  increased  complexity  of  a  distributed  test  can  result  in  a  small  to  medium  increase  in 
cost.  However,  actual  costs  will  be  application  specific.  Cost  drivers  include  SE  complexity, 
fidelity  requirements,  the  requirement  for  experienced  personnel,  configuration  management, 
network  interfaces.  Development  of  a  new  or  modified  simulation  to  support  a  distributed 
test  will  increase  costs.  The  MITRE  Corporation  developed  a  distributed  testing  work 
breakdown  structure  (WBS)  in  support  of  JADS.  The  top-level  WBS  is  presented  in  Table 
24  below.  The  ADS  cost  impact  column  is  an  assessment  of  costs  relative  to  a  conventional 
test.  Because  actual  costs  will  be  program  specific,  it  is  difficult  to  quantify  actual  ADS  cost 
impacts.  In  order  to  provide  test  planners  with  some  insight  into  the  impact  ADS  may  have 
on  a  conventional  test,  JADS  subjectively  determined  the  degree  of  impact  a  particular  ADS 
cost  driver  might  have  as  small,  medium  or  high.  The  increased  complexity  of  an  ADS  test 
results  in  a  small  to  medium  increase  in  the  cost  of  most  categories.  The  cost  drivers  under 
the  design  and  development  category  are  simulations  and  interfaces.  If  a  new  or  modified 
simulation  must  be  developed,  the  cost  can  be  medium  to  high.  For  example,  a  significant 
cost  to  the  JADS  ETE  Test  was  the  development  of  a  high-fidelity  representation  of  the  Joint 
STARS  radar  system  (ETE  Test  Phase  1).  The  cost  to  develop  this  simulation  alone  was  $4.2 
million,  which  represents  53  percent  of  the  entire  ETE  test  program  s  cost. 

Table  24.  ADS  Cost  Impacts 


WBS  Element 

ADS  Cost  Impact 

Planning 

Low  increase 

_ sS — _ — 

Concept  Development 

Low  -  medium  increase 

^  *  . - — — — - - - 

Design  and  Development 

Medium  increase 

Installation,  Integration  and  Test 

Medium  increase 

Text  Execution  and  Analysis 

Large  decrease  -  low  increase 

For  a  complete  discussion  of  the  MITRE  WBS  see  JADS  Special  Report  on  the  Costs  and 
Benefits  of  Distributed  Testing  which  can  be  found  at  www.jads.abq.com. 


*  After  1  March  2001  refer  requests  to  the  Joint  Program  Office  Technical  Library,  2001  North 
Beauregard  St.,  Suite  800,  Alexandria,  Virginia  22311. 
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SBA  advocates  that  the  system  developer  deliver  system  representations  and  distributed 
product  descriptions  early  in  the  acquisition  process  and  maintain  and  update  these 
representations  throughout  the  system  life  cycle.  STEP  guidelines  refer  to  a  “maturing  suite 
■  of  models”.  These  approaches  should  reduce  the  initial  cost  of  the  SUT  simulation  to  the 
government  and  greatly  reduce  or  eliminate  the  cost  of  this  simulation  to  testers.  The  cost 
impact  of  a  new  or  modified  interface  based  on  the  JADS  experience  is  judged  to  be  medium 
to  medium-high.  JADS  determined  that  communications  links  were  not  significant  cost 
drivers,  and  that  these  costs  are  expected  to  decrease  in  the  future.  The  cost  impact  on  test 
execution  and  analysis  runs  the  gamut  from  medium  increases  to  large  cost  savings.  The  cost 
of  test  assets  is  directly  related  to  the  fidelity  of  the  asset  representation.  In  most  cases  a  live 
asset  costs  the  most,  a  virtual  asset  less,  and  a  constructive  asset  the  least.  Distributed  testing 
allows  the  tester  to  obtain  the  required  fidelity  at  minimum  cost.  A  related  issue  that  can 
result  in  cost  savings  is  test  asset  availability.  In  this  case,  live  test  assets  are  generally  the 
least  readily  available,  virtual  more,  and  constructive  the  most  available.  Availability  of  test 
assets  drives  the  test  schedule  especially  when  modifications  and  retesting  are  required. 
JADS  identified  several  cost  drivers  during  the  execution  of  the  three  JADS  tests.  These  cost 
drivers  are  listed  to  emphasize  their  importance  in  the  planning  and  execution  of  a  distributed 
test. 

-  SE  complexity,  as  defined  by  the  number  and  types  of  nodes,  interfaces,  and  bandwidth 

requirements,  will  increase  costs  and  risks  to  all  phases  of  the  distributed  portion  of  the 
test  program.  Since  a  major  virtue  of  distributed  testing  is  to  support  the  construction  of 
complex  test  environments,  the  test  planner  must  balance  the  need  for  complexity  against 
the  expense  and  fragility  of  the  test  architecture. 

-  Fidelity  requirements  for  models  can  range  from  simple,  script-driven  models  to  high- 

fidelity,  man-in-the-loop  simulators.  Fidelity  and  costs  are  directly  proportional. 

-  It  was  the  experience  of  JADS  that  test  ranges  and  labs  underestimated  costs  10  to  15 

percent.  It  is  recommended  that  agreements  with  these  organizations  be  formalized  in  a 
statement  of  capabilities,  and  that  they  be  required  to  provide  detailed  cost  estimates. 
The  program  test’s  priority  number  should  be  provided  to  the  program  management 
office’s  (PMO)  T&E  representative.  Additionally,  any  provisions  for  schedule  delay 
penalties  should  be  agreed  to.  (These  penalties  are  one  of  the  costs  of  poor  scheduling.) 

-  Test  ranges,  labs,  and  other  federates  should  have  experienced  staff  trained  in  distributed 

testing  and  HLA  applications.  If  they  do  not  have  this  capability,  costs  for  elements 
associated  with  their  support  will  be  higher. 

-  JADS  found  that  configuration  management  of  hardware  (HW),  software  (SW),  NIUs,  and 

models  was  more  important  to  distributed  testing  than  to  traditional  testing.  A  detailed 
configuration  management  plan  and  implementation  process  should  minimize  this  risk. 

-  The  cost  of  implementing  distributed  testing  can  be  significantly  reduced  if  it  is  planned 

for  within  the  larger  test  program,  i.e.,  while  traditional  testing  approaches  are  being 
developed. 
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-  Architecture  integration  should  be  fully  understood  to  properly  estimate  it  at  the  outset  of 

the  program.  JADS  found  this  activity  to  be  consistently  underestimated. 

-  Legacy  test  ranges  with  out-of-date  equipment  caused  significant  problems  for  the  JADS 

tests.  This  could  be  clarified  through  a  formal  document  from  the  test  range. 

Table  25  reports  the  actual  costs  to  execute  the  three  JADS  tests. 

Table  25.  JADS  Costs 


JADS  Costs 

srr 

LSP 

$  900,000 

LFP 

$1,400,000 

ETE  Test 

Phase  1 

$4,700,000 

Phase  2 

$2,200,000 

Phase  3 

$  700,000 

Phase  4 

$  300,000 

EW  Test 

OAR  Phase 

$2,120,000 

DSM  Phase 

$2,700,000 

ISTF  Phase 

$1,700,000 
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-  Distributed  testing  supports  or  enhances  cost  avoidance  in  several  ways.  Cost  avoidance  can 
be  thought  of  as  unplanned  for  cost  savings.  By  identifying  SUT  problems  earlier,  you  can 
■pot  only  avoid  the  expense  associated  with  fixing  the  problems  later,  but  you  also  avoid  the 
expense  of  failing  a  more  expensive  test  later.  Similarly,  the  ability  to  identify  tactics, 
techniques  and  procedures  (TTP),  training,  data  processing,  and  analysis  problems  earlier  can 
all  avoid  the  cost  of  expensive  OAR  test  failures  later.  The  LFP  of  the  SIT  provided  a  good 
example  of  cost  avoidance  that  could  be  attributed  to  distributed  testing.  The  AMRAAM 
initial  operational  test  and  evaluation  failed  a  simultaneous,  multiple-shot  test  event.  The 
cause  was  an  interoperability  problem  concerning  the  rear  data  link  information  going  from 
the  launch  aircraft  fire  control  radar  to  the  AMRAAM.  This  interoperability  problem  would 
have  been  found  in  a  LFP  ADS  test.  The  costs  that  would  have  been  avoided  include 
numerous  tests  and  missiles  to  demonstrate  that  the  fix  worked;  the  dollars  associated  with 
the  program  delay;  and  the  additional  gray  hairs  for  the  program  manager  and  test  manager. 
Costs  savings  will  be  very  program  specific  and  closely  related  to  the  complexity  of  the  test 
environment  (potential  for  increased  savings)  and  the  number  of  new  simulations  and 
interfaces  that  need  to  be  developed  instead  of  being  reused.  The  potential  for  cost  avoidance 
can  be  judged  to  be  high,  but  the  ability  to  predict  the  amount  for  any  given  program  is  very 
low. 

3.2.3  Personnel 

-  An  experienced  program  manager  or  system  integrator  is  needed  to  oversee  facility 
development,  because  of  the  difficulty  in  coordinating  several  diverse  facilities  to 
successfully  integrate  an  ADS-linked  configuration.  ADS  requires  strong  systems 
integration  and  systems  engineering  expertise.  This  responsibility  is  difficult  to  manage  by 
participants  supplying  items  to  be  integrated.  If  the  sponsor  is  unable  to  provide  the  expertise 
of  a  systems  engineer  and  integrator,  an  independent  source  should  be  used.  Subject  matter 
experience  and  knowledge  of  computers  and  communications  technology  are  essential  for  the 
systems  integrator.  Simulations,  when  used  in  distributed  testing,  should  be  carefully  planned 
and  developed.  The  distributed  test  controller  must  also  be  in  close  communication  with 
each  organization  modifying  simulations  crucial  to  the  test.  The  ETE  Test  Phase  2 
succeeded  because  of  the  relatively  high  degree  of  reliability  of  the  simulations  used  in  the 
test.  If  simulation  execution  problems  had  occurred,  the  ETE  Test  team  had  arranged  for  a 
quick  response  by  the  personnel  needed  to  fix  those  difficulties. 

-  Have  a  centralized  test  control  center  with  test  controllers  who  are  extremely  familiar  with 
the  test  and  network  configuration. 

-  Personnel  involved  in  a  distributed  test  should  understand  the  “big  picture.”  When  people 
are  geographically  separated,  it  becomes  easy  for  them  to  focus  on  their  own  individual 
portion  of  the  test.  When  problems  arise,  personnel  who  understand  the  entire  test  and  the 
overall  network  will  find  solutions  much  faster. 

-  Distributed  tests  require  personnel  distribution.  When  many  distributed  nodes  are  required 
for  the  successful  completion  of  a  test,  personnel  will  need  to  be  located  at  these  nodes.  The 
complexity  and  input  an  individual  node  contributes  should  guide  the  assignment  of 
personnel.  The  ETE  Test  Phase  2  required  several  people  at  the  Fort  Hood,  Northrop 
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Grumman  and  TCAC  nodes;  only  one  person  was  needed  at  the  White  Sands  node.  The  Fort 
Sill  node  used  only  resident  personnel. 

-  Test  controllers  need  to  be  extremely  familiar  with  the  test  and  network  configuration.  THE 
EW  Phase  2  Test  succeeded  partly  because  it  had  an  experienced  test  controller  with  the 
necessary  tools  to  evaluate  problems  and  the  authority  to  make  meaningful  decisions 
regarding  test  problems. 

-  The  number  of  computers,  intricate  execution  procedures,  and  high  number  of  test  events 
performed  sequentially  created  a  very  workload  intensive  environment  at  the  TCAC  and 
other  locations  during  testing  periods.  Manning  requirements  at  the  TCAC,  AFEWES,  and 
ACETEF  involved  14  dedicated  JADS  personnel  during  the  two-week  test  period.  Site 
manning  (three  persons)  at  AFEWES  was  insufficient.  It  was  determined  that  one  additional 
person  would  be  required  for  rotation  among  JETS,  Tactical  Air  Mission  Simulator  (TAMS) 
and  the  simulator  stations.  JADS  reviewed  and  updated  the  site  manning  matrix  for  Phase  3. 
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4.0  Summary 

4.1  Program  Issue  Accomplishment 

The  JADS  charter  focused  on  three  issues:  What  is  the  present  utility  of  ADS,  including  DIS,  for 
T&E;  what  are  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E; 
and  what  are  the  requirements  that  must  be  introduced  into  ADS  systems  if  they  are  to  support  a 
more  complete  T&E  capability  in  the  future.  As  cited  in  paragraph  3.0  in  the  “Lessons  Learned” 
section  of  this  report,  distributed  architectures  can  be  powerful  tools  capable  of  generating  real 
data  on  the  performance  of  a  SUT.  It’s  important  to  remember  that  the  term  “distributed  testing” 
can  appropriately  be  substituted  for  “advanced  distributed  simulation”  in  some  cases. 

4.1.1  Utility 

Roles  for  ADS  can  be  identified  for  both  DT&E  (in  which  the  general  objective  is  to  determine 
the  SUT's  ability  to  meet  specifications  using  preproduction  subsystems)  and  OT&E  (in  which 
the  general  objective  is  to  determine  the  SUT's  ability  to  meet  user's  operational  effectiveness 
and  suitability  requirements  using  production-level  systems  in  realistic  combat  scenarios).  The 
underlying  ADS  technology  can  support  DT&E  and  OT&E  equally  well. 

JADS  concluded  the  technology,  when  properly  applied,  produces  valid  test  data  over  a  broad 
range  of  systems  and  types  of  tests.  JADS  identified  three  broad  areas  of  benefits  to  using  ADS: 

-  The  ability  to  overcome  current  test  limitations 

-  The  ability  to  identify  failure  modes  earlier 

-  The  ability  to  conduct  end-to-end  testing 

Associated  with  these  benefits  to  distributed  testing  is  the  ability  to  impact  program  cost  through 
either  cost  savings  or  cost  avoidance.  JADS  identified  potential  for  saving  costs  in  two  studies 
where  distributed  testing  would  allow  for  a  reduction  in  live  testing  while  providing  a  more 
complete  test  environment.  JADS  demonstrated  cost  avoidance  by  using  distributed  testing  to 
identify  failure  modes  earlier  and  by  using  distributed  testing  to  rehearse  live  missions,  thus 
identifying  potential  live  test  failures. 

4.1.2  Concerns 

JADS  determined  the  primary  concerns  for  implementing  ADS  are  programmatic  rather  than 
technical.  Distributed  testing  is  not  a  "plug  and  play"  technology.  Developing  a  distributed  test 
environment  requires  a  strong  engineering  methodology  and  attention  to  detail.  Additionally, 
scheduling  multiple  facilities  to  support  both  the  integration  and  execution  of  a  test  is  often  very 
challenging.  Costs  remain  a  concern.  Most  existing  facilities  were  not  designed  to  be  linked  and 
require  some  modification,  along  with  appropriate  interfaces,  before  linking  is  possible.  This 
cost  cm  be  significant  but  can  be  amortized  over  all  phases  of  the  acquisition  process. 
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4.1.3  Constraints  and  Limitations 

JADS  identified  four  areas  where  the  application  of  this  technology  is  constrained. 

-  The  lower  limit  of  latency  is  fixed  by  the  speed  of  light,  and  there  are  a  small  percentage  of 
test  programs  that  involve  very  tightly  coupled  interactions  between  the  SUT  and  other 
players  in  which  latency  will  be  a  constraint. 

—  Another  technical  constraint  is  the  ability  to  present  representations  of  synthetic  targets  to 
live  players.  This  is  especially  true  for  presenting  synthetic  targets  to  human  operators  in  a 
live  vehicle  such  as  an  aircraft  or  a  tank.  Its  also  a  problem  for  radio  frequency  and  infrared 
sensors  on  many  live  vehicles. 

-  There  is  a  limited  ability  to  represent  dynamic  environmental  effects  among  live  and  virtual 
players.  Currently  there  are  no  means  to  capture  weather  effects  or  man-made  effects  such  as 
smoke  and  weather  in  real  time  from  a  live  range  and  synchronize  them  with  similar  effects 
in  a  virtual  or  constmctive  simulation. 

—  The  most  prevalent  distributed  testing  constraint  is  the  availability  and  capability  of  current 
simulations.  Distributed  testing  must  rely  on  the  use  of  simulations  that  were  not  necessarily 
designed  to  work  together.  It  most  cases  it  is  more  cost  effective  to  develop  interface  units 
than  it  is  to  modify  existing  simulations. 

4.1.4  Requirements 

JADS  identified  several  requirements  that,  if  delivered,  will  improve  the  capability  to  implement 
this  technology  in  test  and  evaluation. 

—  Open  systems  standards  for  technical  interoperability  between  live,  virtual,  and  constructive 
simulations 

-  Multilevel  security  capabilities 

-  Distributed  test  control  standards 

—  On-call  networking  capability  providing  low  latency,  deterministic  performance 

-  Distributed  testing  expertise  at  test  ranges  and  facilities 

—  Support  from  DoD  test  and  evaluation  and  acquisition  leadership 

4.2  General  Accomplishments 

The  JADS  JTF  developed  four  distributed  test  environments  connecting  distributed  test  facilities, 
live  air  and  ground  nodes,  laboratories,  and  simulation  facilities.  These  environments  were  used 
to  conduct  three  successful  multiphase  distributed  tests  across  three  domains  of  systems: 
precision  guided  munitions,  C4ISR,  and  electronic  warfare.  Testing  was  conducted  using  the 
environments  for  approximately  215  hours  and  an  additional  200  hours  of  integration,  risk 
reduction,  and  test  rehearsals. 

JADS  was  responsible  for  the  production  of  two  important  products.  The  first  was  a  DIS- 
compliant  version  of  the  Army's  interactive  simulation  called  Janus.  JADS  provided  funding  to 
the  TRAC-WSMR  to  make  the  required  improvements  to  Janus.  The  improved  version  of  Janus 
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has  been  used  in  support  of  multiple  tests  and  exercises  in  addition  to  providing  the  battlefield 
environment  for  the  JADS  End-to-End  Test. 

Another  product  developed  as  a  direct  result  of  a  JADS  test  was  the  VSTARS.  It  is  an  emulation 
of  all  the  radar  functionality  from  the  E-8C  portion  of  the  Joint  STARS  system.  VSTARS 
continues  to  be  used  to  support  the  Joint  STARS  T&E  evaluation  program  and  is  planned  to 
become  an  integral  part  of  the  Joint  STARS  Mission  Crew  Training  System.  VSTARS  has  been 
used  to  provide  virtual  Joint  STARS  radar  for  multiple  exercises.  The  internal  synthetic  aperture 
radar  (SAR)  simulation,  Advanced  Radar  Imaging  Emulation  System  (ARIES)  is  being  ported  to 
provide  an  internal  SAR  training  capability  for  the  Army's  common  ground  station  (CGS). 

JADS  also  developed  products  in-house  with  the  help  of  Science  Applications  International 
Corporation  contractors.  The  first  product  was  the  Analysis  Toolbox.  It  is  a  set  of  C++  routines 
integrated  into  a  single  user  interface  that  allows  users  to  perform  near  real-time  and  post-test 
analysis  by  graphically  plotting  test  data  consisting  of  PDUs.  This  product  provided  dynamic 
capabilities  that  did  not  exist  before  JADS.  The  second  product  was  an  RTI  logger  for  HLA 
simulations  that  resides  between  the  federate  software  and  the  application  program  interface 
(API).  This  product  also  filled  an  analysis  void  and  has  been  widely  distributed  by  JADS  for  use 
by  other  organizations. 

4.3  Program  Impact 

The  potential  of  distributed  testing  as  a  feasible  test  tool,  one  with  the  ability  to  overcome  many 
traditional  shortcomings  in  present  day  T&E  methodologies,  is  exciting.  JADS  investigated  this 
potential  and  developed  a  legacy  program  to  ensure  the  vital  information  gets  to  the  proper 
organizations  in  a  format  they  can  understand  and  use. 

The  legacy  of  JADS  was  much  more  than  a  voluminous,  unread  report.  The  legacy  of  the  JADS 
JT&E  covered  a  broad  range  of  issues  for  the  T&E  community.  JADS  has  defined  its  legacy 
program  as  "all  actions  JADS  takes  to  ensure  that  its  products  are  fully  incorporated  into  the  user 
community."  There  were  three  aspects  to  this  effort. 

/.  Educate  the  user  community  and  instill  distributed  testing  into  its  thought  processes.  JADS 
developed  a  training  course  that  was  offered  at  JADS  and  off  site  upon  request.  The  course 
covered  ADS  concepts;  the  potential  benefits  of  using  ADS;  an  overview  of  the  JADS  test 
events;  lessons  learned  from  completed  tests;  and  methodologies  for  assessing  and  using  ADS. 
The  course  described  ADS,  encouraged  thinking  and  planning  processes  that  include  ADS,  and 
included  recommendations  on  how  and  when  it  might  be  used.  More  than  1500  T&E 
professionals  from  DoD,  industry,  and  international  organizations  have  attended  these  courses. 
In  addition,  JADS  used  a  variety  of  media  and  forums  to  spread  the  word  to  the  test  and 
evaluation  community.  This  activity  included  thirty  seven  test  or  special  reports,  a  quarterly 
newsletter,  a  World  Wide  Web  site,  a  variety  of  brochures,  information  booths  at  T&E 
conferences  and  symposia,  technical  papers,  videos,  and  interactive  multimedia  compact  disks. 

2.  Equip  the  user  community  with  the  proper  distributed  testing  knowledge,  procedures,  and 
tools.  JADS  developed  reports,  training  modules,  roadmaps,  checklists,  etc.,  so  that  testers  can 


109 


assess  whether  distributed  testing  is  right  for  them  in  a  particular  application.  JADS  also 
produced  products  so  that,  having  made  the  determination  that  distributed  testing  is  worthwhile 
in  their  situation,  testers  can  develop  plans  to  develop  a  distributed  test  environment  and  conduct 
a  distributed  test.  Procedures  were  developed  for  communication,  network  design,  installation 
and  checkout;  verification,  validation  and  accreditation  (VV&A);  test  control  and  analysis;  and 
security.  Specialized  software  tools  were  developed  for  network  monitoring,  data  collection,  and 
real-time  data  analysis.  Products  developed  span  the  entire  spectrum  of  distributed  testing  from 
evaluation  to  planning  to  execution  and  analysis.  JIADS  included  information  in  a  variety  of 
media  about  the  prudent  uses  of  distributed  testing,  technical  knowledge,  VV&A  strategies, 
pitfalls,  lessons  learned,  and  a  final  interpretation  of  results. 

3.  Institutionalize  the  products  of  the  JADS  JT&E  for  lasting  value.  JADS  worked  with  a 
variety  of  agencies  and  repositories  to  arrange  for  the  long-term  availability  of  JADS  reports  and 
products.  After  1  March  2001  refer  requests  for  information  to  Headquarters  Air  Force 
Operational  Test  and  Evaluation  Center  History  Office  (HQ  AFOTEC/HO),  8500  Gibson  Blvd. 
SE,  Kirtland  Air  Force  Base,  New  Mexico  87117-5558,  or  SAIC  Technical  Library,  2001  North 
Beauregard  St.,  Suite  800,  Alexandria,  Virginia  22311.  In  this  way,  future  T&E  professionals 
can  access  what  was  learned  and  reap  the  benefits  long  after  JADS  has  ceased  to  exist. 
Additionally,  as  experience  in  the  use  of  distributed  testing  as  a  testing  tool  proliferates,  future 
efforts  may  delve  further  into  this  new  technology.  The  groundbreaking  work  of  JADS  will  then 
be  available  as  a  starting  point  for  further  study. 

Perhaps  more  importantly,  JADS  has  a  variety  of  intangible  products.  Some  of  these  products 
include  knowledge  and  experience  gained  by  T&E  facility  personnel  as  a  result  of  the  tests; 
infrastructure  and  computing  power  paid  for  by  JADS  but  distributed  to  the  appropriate  facilities 
upon  completion  of  tests;  increased  willingness  of  testing  professionals  to  consider  distributed 
testing  as  a  possible  solution  to  their  testing  challenges;  and  the  tools  to  evaluate  distributed 
testing  for  how  it  may  fit  a  particular  application. 

The  impact  of  JADS  will  be  real  change,  where  warranted,  and  the  knowledge  and  tools  needed 
to  implement  those  changes  for  better  T&E  in  the  future.  Better  T&E  can  mean  T&E  at  lower 
cost;  more  complete  T&E  at  the  same  cost;  higher  cost  but  greatly  enhanced  fidelity;  or  in  some 
cases,  the  only  way  to  test  because  of  safety  and/or  environmental  constraints.  Better  T&E 
through  the  intelligent  use  of  distributed  testing  is  all  of  these  and  more.  Giving  our  warfighters 
the  best  we  can  possibly  give  them  is  our  ultimate  goal.  The  proper  use  of  distributed  testing 
will  help  create  weapon  systems  with  lower  overall  life-cycle  costs  that  come  from  better  design, 
testing,  and  evaluation  before  being  put  into  the  hands  of  our  warfighters.  This  is  the  true  legacy 
of  the  JADS  JT&E. 
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5.0  Definitions 


A  ■ 

Accreditation.  See:  distributed  simulation  accreditation,  model/simulation  accreditation. 
Accuracy.  The  degree  of  exactness  of  a  model  or  simulation  relative  to  an  established  standard 
with  high  accuracy  implying  low  error.  [DIS] 

Activity.  An  event  that  consumes  time  and  resources  and  whose  performance  is  necessary  for  a 
system  to  move  from  one  event  to  the  next.  [DIS] 

Advanced  Distributed  Simulation  (ADS).  A  set  of  disparate  models  or  simulations  operating  in  a 
common  synthetic  environment.  The  ADS  may  be  composed  of  three  modes  of  simulation: 
live,  virtual  and  constructive,  where  the  latter  can  be  seamlessly  integrated  within  a  single 
exercise.  See  also:  live  simulation;  virtual  simulation;  constructive  simulation.  [DIS] 
Aggregate.  An  activity  that  combines  individual  entities  into  a  singular  entity.  Contrast  with; 

disaggregate.  [DIS] 

B 

Battlespace.  The  three-dimensional  battlefield.  [DIS] 

Benchmark,  (v)  The  activity  of  comparing  the  results  of  a  model  or  simulation  with  an  accepted 
representation  of  the  process  being  modeled,  (n)  The  accepted  representation  of  the  modeled 
process.  [DIS] 

Bit.  The  smallest  unit  of  information  in  the  binary  system  of  notation.  [DEEE  1278.1] 

Broadcast.  A  transmission  mode  in  which  a  single  message  is  sent  to  all  network  destinations, 
i.e.,  one-to-all.  Broadcast  is  a  special  case  of  multicast.  Contrast  with:  multicast;  unicast. 
[IEEE  1278.2] 

C 

Compatible.  Two  or  more  simulations  are  DIS  compatible  if  (1)  they  are  DIS  compliant,  and  (2) 
their  models  and  data  that  send  and  interpret  PDUs  support  the  realization  of  a  common 
operational  environment  among  the  systems  (coherent  in  time  and  space).  Contrast  with: 
compliant,  interoperable.  [DIS] 

Compliant.  A  simulation  is  DIS  compliant  if  it  can  send  or  receive  PDUs  in  accordance  with 
IEEE  Standard  1278  and  1278  (working  drafts).  A  specific  statement  must  be  made 
regarding  the  qualifications  of  each  PDU.  Contrast  with:  compatible,  interoperable.  [DIS] 
Conceptual  Model.  A  description  of  the  content  and  internal  representations  which  are  the  user’s 
and  developer's  combined  concepts  of  the  exercise.  It  includes  logic  and  algorithms  and 
explicitly  recognizes  assumptions  and  limitations.  [DIS] 

Constructive  Simulation.  Models  and  simulations  that  involve  simulated  people  operating 
simulated  systems.  See  Also:  war  games;  higher  order  model  (HOM).  [DIS] 

Continuous  Model.  (1)  A  mathematical  or  computational  model  whose  output  variables  change 
in  a  continuous  manner;  that  is,  in  changing  from  one  value  to  another,  a  variable  can  take  on 
all  intermediate  values.  For  example,  a  model  depicting  the  rate  of  air  flow  over  an  airplane 
wing.  Syn:  continuous-variable  model.  (2)  A  model  of  a  system  that  behaves  in  a  continuous 
manner.  Contrast  with;  discrete  model.  [DIS] 

Continuous  Simulation.  A  simulation  that  uses  a  continuous  model.  [DIS] 

Continuous-Variable  Model.  See:  continuous  model.  [DIS] 
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Control  Station.  (1)  A  facility  which  provides  the  individual  responsible  for  controlling  the 
simulation  and  the  capability  to  implement  simulation  control  as  protocol  data  units  (PDUs) 
on  the  distributed  interactive  simulation  (DIS)  network. 

Syn:  simulation  -  management  station.  [DIS] 

D 

Data.  Representation  of  facts,  concepts,  or  instructions  in  a  formalized  manner  suitable  for 
communication,  interpretation  or  processing  by  humans  or  automatic  means.  [DIS] 

Database.  A  collection  of  data  organized  according  tp  a  schema  to  serve  one  or  more 
applications.  [DIS] 

Data  Certification.  The  determination  that  data  have  been  verified  and  validated.  (l)Data 
producer  certification  is  the  determination  by  the  data  producer  that  data  have  been  verified 
and  validated  against  documented  standards  of  criteria.  (2)  Data  user  certification  is  the 
determination  by  the  application  sponsor  or  designated  agent  that  data  have  been  verified  and 
validated  as  appropriate  for  the  specific  M&S  usage.  [DIS] 

Data  Logger.  A  device  that  accepts  protocol  data  units  (PDUs)  from  the  network  and  stores 
them  for  later  replay  in  the  same  time  sequence  as  the  PDUs  were  originally  received.  See 
also:  protocol  data  unit  (PDU).  [IEEE  1278.3] 

Data  Validation.  The  documented  assessment  of  data  by  subject  area  experts  and  comparison  to 
known  or  best-estimate  values.  (1)  Data  producer  validation  is  that  documented  assessment 
within  stated  criteria  and  assumptions.  (2)  Data  user  validation  is  that  documented 
assessment  of  data  as  appropriate  for  use  in  an  intended  M&S.  [DIS] 

Data  Verification.  The  use  of  techniques  and  procedures  to  ensure  that  data  meet  specified 
constraints  defined  by  data  standards  and  business  rules.  (1)  Data  producer  verification  is  the 
use  of  techniques  and  procedures  to  ensure  that  data  meet  constraints  defined  by  data 
standards  and  business  rules  derived  from  process  and  data  modeling.  (2)  Data  user 
verification  is  the  use  of  techniques  and  procedures  to  ensure  that  data  meet  user  specified 
constraints  defined  by  data  standards  and  business  rules  derived  from  process  and  data 
modeling  and  that  data  are  transformed  and  formatted  properly.  [DIS] 

Data  Verification,  Validation,  and  Certification.  The  process  of  verifying  the  internal 
consistency  and  correctness  of  data,  validating  that  they  represent  real  world  entities 
appropriate  for  their  intended  purpose  or  an  expected  range  of  purposes,  and  certifying  them 
as  having  a  specified  level  of  quality  or  as  being  appropriate  for  a  specified  use,  type  of  use, 
or  range  of  uses.  The  process  has  two  perspectives:  producer  and  user  process.  See:  data 
validation,  data  verification,  and  data  certification.  [DIS] 

Dead  Reckoning.  See:  remote  entity  approximation. 

Deaggregate.  See:  disaggregate.  [DIS] 

Distributed  Interactive  Simulation  (DIS).  A  synthetic  environment  within  which  humans  may 
interact  through  simulation(s)  at  multiple  sites  networked  using  compliant  architecture, 
protocols,  standards,  and  databases  (DoDD  5000.59P) 

Digital  System  Model  (DSM).  A  digital  representation  of  an  actual  or  conceptual  system  that 
involves  mathematics,  logical  expressions,  or  computer  simulations  that  can  be  used  to  predict 
how  the  system  might  perform  or  survive  under  various  conditions  or  in  a  range  of  hostile 
environments. 

E 

Electronic  Battlefield.  See:  synthetic  environment.  [DIS] 
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Entity.  Any  component  in  a  system  that  requires  explicit  representation  in  a  model.  Entities 
possess  attributes  denoting  specific  properties.  See:  simulation  entity.  [DIS] 

Environment.  (1)  The  texture  or  detail  of  the  domain,  such  as  cities,  farmland,  sea  states,  etc.  (2) 
■The  external  objects,  conditions,  and  processes  that  influence  the  behavior  of  a  system  (such 
as  terrain  relief,  weather,  day,  night,  terrain  cultural  features,  etc.)  [DIS] 

Event.  (1)  An  occurrence  that  causes  a  change  of  state  in  a  simulation.  See  also:  conditional 
event;  time-dependent  event.  (2)  The  instant  in  time  at  which  a  change  in  some  variable 
occurs.  [DIS] 

Event-Driven  Simulation.  See:  event-oriented  simulation.  [DIS] 

Event-Oriented  Simulation.  A  simulation  in  which  attention  is  focused  on  the  occurrence  of 
events  and  the  times  at  which  those  events  occur;  for  example,  a  simulation  of  a  digital 
circuit  that  focuses  on  the  time  of  state  transition.  Syn:  event-driven  simulation;  event- 
sequenced  simulation.  [DIS] 

Event-Sequenced  Simulation.  See:  event-oriented  simulation.  [DIS] 

Exercise.  (1)  One  or  more  sessions  with  a  common  objective  and  accreditation.  (2)  The  total 
process  of  designing,  assembling,  testing,  conducting,  evaluating,  and  reporting  on  an 
activity.  See:  simulation  exercise.  Syn:  experiment,  demonstration.  [DIS,  IEEE  1278.3] 

F 

Fidelity.  (1)  The  similarity,  both  physical  and  functional,  between  the  simulation  and  that  which 
it  simulates.  (2)  A  measure  of  the  realism  of  a  simulation.  (3)  The  degree  to  which  the 
representation  within  a  simulation  is  similar  to  a  real-world  object,  feature,  or  condition  in  a 
measurable  or  perceivable  manner.  See  also:  model/simulation  validation.  [DIS,  IEEE 
1278.1] 

Field.  (1)  A  series  of  contiguous  bits,  treated  as  an  instance  of  a  particular  data  type,  that  may  be 
part  of  a  higher  level  data  structure.  (2)  An  external  operating  area  for  actual  vehicles  or  live 
entities.  See:  field  instrumentation.  [DIS,  IEEE  1278.1] 

G 

Graphical  Model.  A  symbolic  model  whose  properties  are  expressed  in  diagrams.  For  example, 
a  decision  tree  used  to  express  a  complex  procedure.  Contrast  with:  mathematical  model; 
narrative  model;  software  model;  tabular  model.  [DIS] 

Ground  Truth.  The  actual  facts  of  a  situation  without  errors  introduced  by  sensors  or  human 
perception  and  judgment.  [DIS] 

H 

Human-in-the-Loop  Model.  See:  interactive  model. 

Human-Machine  Simulation.  A  simulation  carried  out  by  both  human  participants  and 

computers,  typically  with  the  human  participants  asked  to  make  decisions  and  a  computer 
performing  processing  based  on  those  decisions.  [DIS] 

I 

Interactive  Model.  A  model  that  requires  human  participation.  Syn:  human-in-the-loop  model. 
[DIS] 

Interoperable.  Two  or  more  simulations  are  DIS  interoperable  for  a  given  exercise  if  they  are 
DIS  compliant,  DIS  compatible,  and  their  performance  characteristics  support  a  fair  fight  to 
the  fidelity  required  for  the  exercise.  Contrast  with:  compatible,  compliant.  [DIS] 
Interoperability.  (1)  The  ability  of  a  set  of  simulation  entities  to  interact  with  an  acceptable 
degree  of  fidelity.  The  acceptability  of  a  model  is  determined  by  the  user  for  the  specific 
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purpose  of  the  exercise,  test,  or  analysis.  (2)  The  ability  of  a  set  of  distributed  interactive 
simulation  applications  to  interact  through  the  exchange  of  protocol  data  units.  [DIS] 

L 

Live  Entity.  A  perceptible  object  that  can  appear  in  the  virtual  battlespace  but  is  unaware  and 
non-responsive  (either  by  intent,  lack  of  capability  or  circumstance)  to  the  actions  of  virtual 
entities.  See  also:  field  instrumentation.  Contrast  with:  live  instrumented  entity.  [DIS] 

Live  Instmmented  Entity.  A  physical  entity  that  is  in  the  real  world  and  can  be  represented  in 
the  distributed  interactive  simulation  (DIS)  virtual  battlespace  which  can  be  manned  or 
unmanned.  The  live  instrumented  entity  has  internal  and/or  external  field  instrumentation 
(FI)  devices/systems  to  record  and  relay  the  entity’s  surroundings,  behavior,  and/or  reaction 
to  events.  If  the  FI  provides  a  two-way  link,  the  events  that  affect  the  live  instrumented 
entity  can  be  occurring  in  the  virtual  battlespace  as  well  as  the  real  world.  See  also:  field 
instrumentation,  live  entity.  [DIS] 

Local  Area  Network  (LAN).  A  class  of  data  network  which  provides  high  data  rate 
interconnection  between  network  nodes  in  close  physical  proximity.  [IEEE  1278.3] 

M 

Measure  of  Performance  (MOP).  Measure  of  how  the  system/individual  performs  its  functions 
in  a  given  environment  (e.g.,  number  of  targets  detected,  reaction  time,  number  of  targets 
nominated,  susceptibility  of  deception,  task  completion  time).  It  is  closely  related  to  inherent 
parameters  (physical  and  structural)  but  measures  attributes  of  system  behavior.  See  also: 
measures  of  effectiveness  (MOE).  [DEE  1278.3] 

Model.  (1)  An  approximation,  representation,  or  idealization  of  selected  aspects  of  the  structure, 
behavior,  operation,  or  other  characteristics  of  a  real-world  process,  concept,  or  system. 
Note:  Models  may  have  other  models  as  components.  (2)  To  serve  as  a  model  as  in  (1).  (3) 
To  develop  or  use  a  model  as  in  (1).  (4)  A  mathematical  or  otherwise  logical  representation 
of  a  system  or  a  system’s  behavior  over  time.  [DIS] 

Model/Simulation  Accreditation.  The  official  certification  that  a  model  or  simulation  is 
acceptable  for  use  for  a  specific  purpose.  See  also:  distributed  simulation  accreditation. 
Contrast  with:  model/simulation  validation,  model/simulation  verification.  [DoDD  5000.59] 
Model/Simulation  Validation.  The  process  of  determining  the  degree  to  which  a  model  is  an 
accurate  representation  of  the  real  world  from  the  perspective  of  the  intended  use(s)  of  the 
model.  See  also:  distributed  simulation  validation,  fidelity.  Contrast  with:  model  simulation 
accreditation,  model  simulation  verification.  [DoDD  5000.59] 

Model/Simulation  Verification.  The  process  of  determining  that  a  model  implementation 
accurately  represents  the  developer's  conceptual  description  and  specifications.  See  also: 
distributed  simulation  verification.  Contrast  with:  model  simulation  accreditation,  model 
simulation  validation.  [DoDD  5000.59] 

N 

Network  Filter.  A  system  to  selectively  accept  or  reject  data  received  from  the  network.  [DIS] 
Network  Node.  A  specific  network  address.  See:  node.  Contrast  with:  processing  node.  [DIS] 
Node.  A  general  term  denoting  either  a  switching  element  in  a  network  or  a  host  computer 
attached  to  a  network.  See:  processing  node;  network  node.  [IEEE  1278.1,  IEEE  1278.2] 

O 

Operational  Environment.  A  composite  of  the  conditions,  circumstances,  and  influences  which 
affect  the  employment  of  military  (or  other)  forces  and  the  decisions  of  the  unit  commander 
or  person  in  charge.  [DIS] 
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p 

Platform.  A  generic  term  used  to  describe  a  level  of  representation  equating  to  vehicles,  aircraft, 
missiles,  ships,  fixed  sites,  etc.,  in  the  hierarchy  of  representation  possibilities.  Other 
■.representation  levels  include  units  (made  up  of  platforms)  and  components  or  modules 
(which  make  up  platforms.)  [DIS] 

Protocol  Data  Unit  (PDU).  A  DIS  data  message  that  is  passed  on  a  network  between  simulation 
applications  according  to  a  defined  protocol.  [IEEE  1278.1] 

R 

Real  Time.  In  modeling  and  simulation,  simulated  time  advances  at  the  same  rate  as  actual  time; 
for  example,  running  the  simulation  for  one  second  results  in  the  model  advancing  time  by 
one  second.  Contrast  with:  fast  time,  slow  time.  [DIS] 

Resolution.  (1)  The  degree  to  which  near  equal  results  values  can  be  discriminated.  (2)  The 
measure  of  the  ability  to  delineate  picture  detail.  [DIS] 

S 

Scenario.  (1)  Description  of  an  exercise  (initial  conditions).  It  is  part  of  the  session  database 
which  configures  the  units  and  platforms  and  places  them  in  specific  locations  with  specific 
missions.  (2)  An  initial  set  of  conditions  and  time  line  of  significant  events  imposed  on 
trainees  or  systems  to  achieve  exercise  objectives.  See:  field  exercise.  [DIS,  IEEE  1278.3] 

SIMNET  (Simulator  Networking).  The  prototype  distributed  simulation  upon  which  DIS  was 
based.  [DIS] 

Simulate.  To  represent  a  system  by  a  model  that  behaves  or  operates  like  the  system.  See  also: 
emulate.  [DIS] 

Simulated  Time.  Time  as  represented  within  a  simulation.  Syn:  virtual  time.  See  also:  fast  time; 
real  time;  slow  time.  [DIS] 

Simulation.  (1)  A  model  that  behaves  or  operates  like  a  given  system  when  provided  a  set  of 
controlled  inputs.  Syn:  simulation  model.  See  also:  emulation.  (2)  The  process  of  developing 
or  using  a  model  as  in  (1).  (3)  An  implementation  of  a  special  kind  of  model  that  represents 
at  least  some  key  internal  elements  of  a  system  and  describes  how  those  elements  interact 
over  time.  [DIS] 

Simulation  Environment.  (1)  Consists  of  the  natural  physical  environment  surrounding  the 
simulation  entities  including  land,  oceans,  atmosphere,  near-space,  and  cultural  information. 
(2)  All  the  conditions,  circumstances,  and  influences  surrounding  and  affecting  simulation 
entities  including  those  stated  in  (1).  [DIS] 

Simulation  Exercise.  An  exercise  that  consists  of  one  or  more  interacting  simulation 

applications.  Simulations  participating  in  the  same  simulation  exercise  share  a  common 
identifying  number  called  the  exercise  identifier.  These  simulations  also  utilize  correlated 
representations  of  the  synthetic  environment  in  which  they  operate.  See:  live  simulation. 
[IEEE  1278.1,  IEEE  1278.2] 

Simulation  Fidelity.  Refers  to  the  degree  of  similarity  between  the  simulated  situation  and  the 
operational  situation.  [IEEE  1278.3] 

Simulation  Time.  (1)  A  simulation’s  internal  representation  of  time.  Simulation  time  may 
accumulate  faster,  slower,  or  at  the  same  pace  as  real  time.  (2)  The  reference  time  (e.g., 
universal  coordinated  time)  within  a  simulation  exercise.  This  time  is  established  ahead  of 
time  by  the  simulation  management  function  and  is  common  to  all  participants  in  a  particular 
exercise.  [DIS,  IEEE  1278.1] 
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Simulator.  ( 1 )  A  device,  computer  program,  or  system  that  performs  simulation.  (2)  For  training, 
a  device  which  duplicates  the  essential  features  of  a  task  situation  and  provides  for  direct 
practice.  (3)  For  distributed  interactive  simulation  (DIS),  a  physical  model  or  simulation  of  a 
weapons  system,  set  of  weapon  systems,  or  piece  of  equipment  which  represents  some  major 
aspects  of  the  equipment’s  operation.  [DIS] 

Site.  (1)  An  actual  physical  location  at  a  specific  geographic  area,  e.g.,  the  Fort  Knox  Close 
Combat  Test  Bed  (CCTB).  (2)  A  node  on  the  network  used  for  distributed  simulation  such 
as  the  Defense  Simulation  Internet  (DSI)  long  haul  network.  (3)  A  level  of  configuration 
authority  within  a  DIS  exercise.  [DIS] 

V 

Validation.  See:  data  validation,  distributed  simulation  validation,  face  validation, 
model/simulation  validation.  [DIS] 

Verification.  See:  data  verification,  distributed  simulation  verification,  model/simulation 
verification 

Verification  and  Validation  (V&V)  Proponent.  The  agency  responsible  for  ensuring  V&V  is 
performed  on  a  specific  model  or  simulation.  [DIS] 

Vignette.  A  self-contained  portion  of  a  scenario.  [DIS] 

Virtual  Battlespace.  The  illusion  resulting  from  simulating  the  actual  battlespace.  [DIS] 

W 

War  Game.  A  simulation  game  in  which  participants  seek  to  achieve  a  specified  military 

objective  given  pre-established  resources  and  constraints;  for  example,  a  simulation  in  which 
participants  make  battlefield  decisions  and  a  computer  determines  the  results  of  those 
decisions.  See  also:  management  game.  Syn:  constructive  simulation;  higher  order  model 
(HOM).  [DIS] 

Wide  Area  Network  (WAN).  A  communications  network  of  devices  which  are  separated  by 
substantial  geographical  distance.  Syn:  long  haul  network.  [IEEE  1278.3] 
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6.0  Acronyms 


A/e 

A^ATD 

AASI 

ACE 

ACETEF 

ADEWS 

ADRS 

ADS 

ADT 

AFATDS 

AFB 

AFEWES 

AFOTEC 

AIM 

ALQ-131 


AMRAAM 

API 

ARIES 

ASAS 

ATACMS 

AVTB 

BMIC 


C4ISR 


CCF 

CGS 

CONOPS 

CROSSBOW 


CSU 

DIS 

DMAP 

DMSO 

DoD 

DSI 

DSM 


aircraft 

Anti-Armor  Advanced  Technology  Demonstration 
Advanced  Aircraft  Simulation  Interface 
analysis  and  control  element 

Air  Combat  Environment  Test -and  Evaluation  Facility,  Patuxent  River, 
Maryland;  Navy  facility 

Advanced  Distributed  Electronic  Warfare  System;  Army  sponsored 
Automated  Data  Reduction  Software 
advanced  distributed  simulation 
air  data  terminal 

Advanced  Field  Artillery  Tactical  Data  System 
Air  Force  Base 

Air  Force  Electronic  Warfare  Evaluation  Simulator,  Fort  Worth,  Texas; 

Air  Force  managed  with  Lockheed  Martin  Corporation 

Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air  Force 

Base,  New  Mexico 

air  intercept  missile 

a  mature  self-protection  jammer  system;  an  electronic  countermeasures 
system  with  reprogrammable  processor  developed  by  Georgia  Tech 
Research  Institute 

advanced  medium  range  air-to-air  missile 

application  program  interface 

Advanced  Radar  Imaging  Emulation  System 

All  Source  Analysis  System 

Army  Tactical  Missile  System 

Aviation  Test  Bed  at  Fort  Rucker,  Alabama 

Battle  Management  Interoperability  Center  at  Naval  Air  Warfare  Center, 
Point  Mugu,  California 

command,  control,  communications,  computers  and  intelligence 
command,  control,  communications,  computers,  intelligence,  surveillance 
and  reconnaissance 

Central  Control  Facility,  Eglin  Air  Force  Base,  Florida 
common  ground  station 
concept  of  operations 

Office  of  the  Secretary  of  Defense  committee  under  the  Director, 

Operational  Test  and  Evaluation 

channel  service  unit 

distributed  interactive  simulation 

data  management  and  analysis  plan 

Defense  Modeling  and  Simulation  Organization,  Alexandria,  Virginia 
Department  of  Defense 
Defense  Simulation  Network 
digital  system  model 
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DSU 

data  service  unit 

DT&E 

developmental  test  and  evaluation 

ECM 

electronic  countermeasures 

EPE 

engineering  protofederation 

ES 

electronic  support 

ESPDU 

entity  state  protocol  data  unit 

ETE 

JADS  End-to-End  Test 

EW 

electronic  warfare;  JADS  Electronic  Warfare  Test 

FAT 

federate  acceptance  test 

FBCB^ 

Force  XXI  Battle  Command,  Brigade  and  Below 

FED 

federation 

FEDEP 

federation  development  and  execution  process 

FEDEX 

federation  executive 

FIT 

federate  integration  test 

FOM 

federation  object  model 

FOT&E 

follow-on  operational  test  and  evaluation 

FTP 

file  transfer  protocol 

FY 

fiscal  year 

GB 

gigabyte 

GDT 

ground  data  terminal 

GPS 

global  positioning  system 

HITE 

hardware-in-the-loop  (electronic  warfare  references) 

HLA 

high  level  architecture 

HW 

hardware 

HWIL 

hardware-in-the-loop  (system  integration  references) 

IADS 

Integrated  Air  Defense  System 

ICD 

interface  control  document 

ID 

infantry  division;  identification 

IDNXTM 

Integrated  Digital  Network  Exchange 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

IGMP 

Internet  Group  Management  Protocol 

INS 

inertial  navigation  system 

IP 

Internet  protocol 

IPPD 

integrated  product  and  process  development 

IPT 

integrated  product  team 

IR 

infrared 

IRIG 

Inter-Range  Instrumentation  Group 

IRIX 

operating  system  for  the  Silicon  Graphics,  Inc. 

ISTF 

installed  systems  test  facility 

J/S 

jamming-to-signal  ratio 

JADS 

Joint  Advanced  Distributed  Simulation,  Albuquerque,  New  Mexico 

Janus 

interactive,  computer-based  simulation  of  combat  operations 

JCSAR 

Joint  Combat  Search  and  Rescue 

JECSIM 

Joint  Electronic  Combat  Testing  Using  Digital  Simulations 

JETS 

JammEr  Techniques  Simulator 

Joint  STARS 

Joint  Surveillance  Target  Attack  Radar  System 

118 


JSF 

JT&E 

JTF 

JTMD 

LAN 

LFP 

LGSM 

LHC 

LRC 

LSP 

M&S 

Mbps 

MCTS 

MISILAB 

MITRE 

MOE 

MOP 

MOT&E 

MSL 

NATO 

NETVisualizeH^ 


NIU 

NTP 

OAR 

OPTEMPO 

OSD 

OT 

OT&E 

OTA 

PC 

PDU 

PGM 

PIM-DM 

PMO 

P-value 

RDAPAS 

RDL 

RELDISTR 

RF 

RFENV 

RID 

RM&A 

ROE 

RTI 

RTIEXEC 


Joint  Strike  Fighter 
joint  test  and  evaluation 
joint  test  force 

Joint  Theater  Missile  Defense 
local  area  network 
Live  Fly  Phase 
light  ground  station  module 
link  health  check 

local  runtime  infrastructure  component 
Linked  Simulators  Phase 
modeling  and  simulation 
megabits  per  second 
Mission  Crew  Training  System 

Missile  Simulation  Laboratory,  Eglin  Air  Force  Base,  Florida 

company  that  provided  engineering  services 

measure  of  effectiveness 

measure  of  performance 

multiservice  operational  test  and  evaluation 

missile 

North  Atlantic  Treaty  Organization 

software  that  displays  real-time  bandwidth  use  in  a  rolling  bar  graph  format 

for  quick  visual  reference 

network  interface  unit 

network  time  protocol 

open  air  range 

operations  tempo 

Office  of  the  Secretary  of  Defense 

operational  test 

operational  test  and  evaluation 

operational  test  agency 

personal  computer 

protocol  data  unit 

precision  guided  munitions 

protocol  independent  multicast-dense  mode 

program  management  office 

probable  value 

Radar  Detection  and  Performance  Analysis  System 

rear  data  link 

reliable  distribution 

radio  frequency 

radio  frequency  environment 

runtime  infrastructure  initialization  data 

reliability,  maintainability  and  availability 

rule  of  engagement 

runtime  infrastructure 

runtime  infrastmcture  executive 
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SAIC 

SAR 

SATCOM 

SBA 

SCDL 

SE 

SEOT 

SETI 

SGI 

SIL 

SM 

SMLAB 

SINCGARS 

srr 

SMC 

SME 

SMS 

SOW 

SPECTRUM® 

SPJ 

SRS 

STEP 

STORM 

STTAR 

SUT 

SW 

T&E 

T/E 

T-1 


T-3 

TAC 

TACCSF 

TAFSM 

TAMS 

TCAC 

TCP 

TCP 

TDP 

TEMP 

TGT 

TMD 

TRAC 

TSLA 

TSPI 

TTH 


Science  Applications  International  Corporation 

synthetic  aperture  radar 

satellite  communications 

Simulation  Based  Acquisition 

surveillance  control  data  link 

synthetic  environment 

synthetic  environment  operational  test 

Synthetic  Environment  Tactical  Integration 

Silicon  Graphics,  Inc. 

system  integration  laboratory 

simulation  . 

Simulation  Laboratory,  Naval  Air  Warfare  Center,  China  Lake,  California 

Single-Channel  Ground  and  Airborne  Radio  System 

IADS  System  Integration  Test 

source  mode  change 

subject  matter  experts 

stores  management  system 

statement  of  work 

a  network  analysis  package  developed  by  Cabletron  Systems 

self-protection  jammer 

software  requirements  specification 

simulation,  test  and  evaluation  process 

Simulation,  Testing  and  Operations  Rehearsal  Model 

synthetic  test  and  training  architecture 

system  under  test 

software 

test  and  evaluation 
tracking  error 

digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1.544  megabits 
per  second 

28  T-1  lines  in  one;  the  aggregate  data  rate  is  44.746  megabits  per  second 
target  analysis  cell 

Theater  Air  Command  and  Control  Simulation  Facility 
Tactical  Army  Fire  Support  Model 
Tactical  Air  Mission  Simulator 

Test  Control  and  Analysis  Center,  Albuquerque,  New  Mexico 
test  control  federate 
transmission  control  protocol 

time-space-position  information  data  optimizing  processor 

test  and  evaluation  master  plan 

target 

Theater  Missile  Defense 

U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  Analysis  Center 
Threat  Simulator  Linking  Activity 
time-space-position  information 
terminal  threat  hand-off  federate 
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TIP 

tactics,  techniques  and  procedures 

UDP 

user  data  protocol 

UMB 

umbilical 

UNIXTM 

registered  trademark  of  UNIX  Systems  Laboratories 

V&V 

verification  and  validation 

VPG 

virtual  proving  ground 

VSTARS 

Virtual  Surveillance  Target  Attack  Radar  System 

VTP 

Virtual  Torpedo  Project 

VV&A 

verification,  validation,  and  accreditation 

WAN 

wide  area  network 

WBS 

work  breakdown  structure 

WSIC 

Weapons  System  Integration  Center,  Naval  Air  Warfare  Center,  Point 
Mugu,  California 

WSMR 

White  Sands  Missile  Range,  New  Mexico 

WSSF 

Weapon  System  Support  Facility,  China  Lake,  California 
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Black,  August  1998 

JADS  Electronic  Warfare  Baseline  Testing,  Maj  Darrell  Wright,  June  1999 

JADS  End-to-End  Test  -  Distributed  Simulation  Using  Satellites,  Gary  Marchand,  September  1999 

JADS  End-to-End  Test — The  Final  Chapter,  Gary  Marchand,  December  1999 

Joint  Test  Environment  for  Electronic  Warfare  Testing,  Maj  Mark  McCall,  December  1997 

Latency  and  its  Effects  on  the  Fidelity  of  Air-to-Air  Missile  T&E  Using  Advanced  Distributed 
Simulations,  Dr  Larry  McKee,  September  1997 

Modification  of  the  Entity  State  PDU  for  Use  in  the  End-to-End  Test,  MAJ  Terry  Schmidt  and  Gary 
Marchand,  December  1996 

Network  Design  and  Performance  of  the  System  Integration  Test,  Linked  Simulators  Phase,  Gregory 
Grundhoffer,  David  Brown,  and  TSgt  Charles  Ashton,  December  1996 

Statistical  Techniques  for  Determining  the  Repeatability  of  Man-in-the-Loop  System  Performance  Data, 
Capt  Sandra  Smith,  Capt  Roman  Nation  and  Maj  Darrell  Wright,  December  1999 

Testing  Advanced  Distributed  Simulation  for  Use  in  Electronic  Warfare  Test  and  Evaluation,  Maj  Darrell 
Wright  and  Clyde  Harris,  September  1997 

Testing  and  Training  in  a  Command,  Communications,  Control,  and  Intelligence  (C3I)  Framework,  John 
Reeves,  December  1997 

The  Applicability  of  ADS  to  Precision  Guided  Munitions  Testing,  Dr  Larry  McKee,  September  1998 

The  Benefits  of  Using  Advanced  Distributed  Simulation  for  Air-to-Air  Missile  Testing,  Dr.  Larry  McKee, 
November  1998 

The  Development  of  a  Real  Time  Stochastic  Radar  Simulation  (VSTARS),  Gary  Marchand  and  MAJ  Terry 
Schmidt,  December  1996 

The  Future  of  Digital  Terrain  in  Distributed  Simulations,  Capt  Rodney  Houser,  December  1998 
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The  J ADS  Analysis  Toolbox  (A  Tool  for  Analysis  of  Distributed  Simulations),  Dean  Gonzalez  and  Jerry 
Black,  March  1999 

The  Utility  of  ADS  for  Precision  Guided  Munitions  Testing,  Dr.  Larry  McKee,  September  1998 

Tuning  RTI  Performance  for  an  Electronic  Warfare  T&E  Federation,  Clyde  Harris  and  Jerry  Black, 
December  1998 


Using  High  Level  Architecture  for  Electronic  Warfare  Test  and  Evaluation,  Clyde  Hams,  November 

1998 

Validation  of  Air-to-Air  Missile  Performance  in  Advanced  Distributed  Simulations,  Dr  Larry  McKee,  July 
1997 

Verification  and  Validation  of  Distributed  Air-to-Air  Missile  Tests,  Dr.  Larry  McKee,  March  1999 

Verification  and  Validation  of  the  JADS  End-to-End  Test-The  Final  Chapter,  Gary  Marchand,  December 

1999 

VSTARS—A  STEP  Success  Story,  Gary  Marchand,  December  1998 

7.8  Tools 

Distributed  Testing  -  A  Tool  for  the  Tester’s  Toolbox  (compact  disk).  This  CD  presents  the  JAJDS 
training  course  including  sections  on 

•  Why  Distributed  Testing 

•  Technology  Concepts  of  Distributed  Testing  (DIS  and  HLA) 

•  Applications  of  Distributed  Testing  (PGM,  C4ISR,  and  EW  testing) 

•  Test  Concept  Development 

•  Distributed  Test  Planning  and  Execution 

•  Methodologies  and  Special  Topics  (V&V,  cost  analysis,  test  control,  terrain/feature  database,  RTI 
performance  testing,  networking,  programmatic  challenges,  time  synchronization  and  security) 

JADS  Analysis  Toolbox.  Dean  Gonzalez  and  Jerry  Black. 

The  JADS  Analysis  Toolbox  Users  Manual,  Dean  Gonzalez  and  Jerry  Black,  June  1999 
The  toolbox  comprises  a  set  of  C++  routines  integrated  into  a  single  user  interface.  Users  can  view 
tabulations  and  plots  of  distributed  interactive  simulation  protocol  data  units  data  in  near  real  time,  can 
replay  or  get  selected  data  in  a  text-readable  format  from  the  JADS  logfile(s)  post  test,  and  can  obtain 
predefined  plots  and  tabulations  of  PDU  statistics  for  post-test  analyses. 

Runtime  Infrastructure  Logger.  This  is  a  set  of  tools  that  log  messages  to  and  from  the  RTI. 

Web  Site 

http://www.jads.abq.com 

After  1  March  2(X)1,  refer  requests  to  Headquarters  Air  Force  Operational  Test  and  Evaluation  Center 
History  Office  (HQ  AFOTEC/HO),  8500  Gibson  Boulevard  SE,  Kirtland  Air  Force  Base,  New  Mexico 
87117-5558  or  Science  Applications  International  Corporation  (SAIC)  Technical  Library,  2001  North 
Beauregard  Street,  Suite  800,  Alexandria,  Virginia  22311. 
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